Blog Posts by Anupam Seth

  • Groundhog: Hadoop Fork Testing

    Authors: Anupam Seth, Robert Evans, and Thomas Graves, Core-Hadoop development team, Cloud Infrastructure Group.

    1. Introduction

    Hadoop is widely used at Yahoo! to do all kinds of processing. It is used for everything from counting ad clicks to optimizing what is shown on the front page for each individual user. Deploying a major release of Hadoop to all 40,000+ nodes at Yahoo! is a long and painful process that impacts all users of Hadoop. It involves doing a staged rollout onto different clusters of increasing importance (e.g. QA, sandbox, research, production) and asking all teams that use Hadoop to verify that their applications work with this new version. This is to harden the new release before it is deployed on clusters that directly impact revenue, but it comes at the expense of the users of these clusters because they have to share the pain of stabilizing a newer version. Further, this process can take over 6 months. Waiting 6 months to get a new feature, which users have

    Read More »from Groundhog: Hadoop Fork Testing