At the Summit and since it ended, I have received questions from folks who attended the show and some who couldn’t make it. In general, a lot of people were curious about what went into developing the Summit and the approach we took to the Summit. I thought I’d take some time today and summarize my thoughts on this topic.

Obviously, in conference planning, a lot of the success of an event comes down to logistics, and fitting 1,700 people into the Santa Clara Convention Center for one action-packed today definitely requires a lot of logistics. But beyond those details, I think more important this year was the decision to change how we approached the Summit and to make sure the focus of the event was on building the Hadoop community itself. The Hadoop community will be at the heart of that continued innovation, so it is important that the community continues to grow and share with each other.

Here is what the Summit was really about for me and what I asked the team to focus on:

1. Content: This year we moved away from the “come as you please” style for presentation content that we had used in years past. What is the line that I used to stress this to the team? “Content is to the Summit what Location is to Real-Estate.”. Everything. Therefore all technical content was first selected then shepherded through a rigorous review and feedback process by the Program Committee. As a result, we heard some fantastic feedback on the quality and usefulness of the presentations this year in the technical tracks. Raymie Stata, Yahoo!’s CTO, made the point numerous times, saying that as a technology grows you usually see the amount and depth of technical content at conferences dedicated to that technology erode, but that at Hadoop Summit was just the opposite. If anything, the technical content was even deeper, which bodes well for Hadoop and the communities future. I cannot thank my Track Co-chairs enough for their support in making this happen. You can find out more about them at: www.hadoopsummit.org.

2. Sponsored Content: Related to keeping the overall quality of technical content high at the Summit was the desire to keep the amount of sponsored content relatively low, ideally in the ball park of 30 minutes out of the day. Mind you, the goal was never not to have any sponsored content. Sponsors who are offering Hadoop-related solutions have a significant role to play in the Hadoop Ecosystem and its evolution, but especially with so many new attendees and folks just getting started with Hadoop, the most valuable content is the technical focused insights they can put to work as they ramp up projects and get up to speed. The results speak for themselves – check out my fun fact number 3 below.

3. Ecosystem: There was some discussion and thinking early on in the conference planning that we should only include open source solutions in the program. However, in the end the majority of the Program Committee agreed that the Summit should represent the entire Hadoop ecosystem. Why? Because while technically it’s possible to separate Apache Hadoop from other Hadoop powered solutions, actual users don’t always make this distinction when putting Hadoop to work. Some use Apache, some use other distributions, but all of these folks are Hadoop users and are therefore part of the Hadoop ecosystem. Without users, there is no Hadoop, so who are we to leave some out?

4. Users: Speaking of users, they are who motivated us to have a brand new track this year focused on them, the Operations and Experience track. The goal of this track was to provide a forum for sharing how different companies are operating and managing Hadoop in the real world, or otherwise talk about their experience with Hadoop. As we had expected, this content was particularly popular with Hadoop users this year. I believe, in the years to come, given the pace of Hadoop growth the interest in this track will continue to increase and the content will no doubt expand as well.

5. Developers: Finally, developers are still the core of the Hadoop Summit and are still the engine for innovation in Hadoop. Many attendees complimented me on what a great “event” Hadoop Summit was. Interestingly, I never really thought of the Summit as an event. For me it was a Hadoop developer and user gathering. As a developer myself, having been there and done that, now I enjoy helping showcase the amazing, high quality work that comes from the Hadoop community of developers. Helping great code get shared and adopted by developers and users is really the heart and soul of the Summit for me.

Looking forward, on the heels of a successful Summit, what’s next? Here at Yahoo! I’ve been saying that Hadoop the Software is maturing, while Hadoop the Product is still in its nascence. Do I mean that Hadoop the Software is done? Not at all. What I mean is that future work on Hadoop as software will be focused on transforming it into Hadoop the product. That means a lot of development on Usability, Manageability and Operability. These very broad areas cover a lot of ground, but collectively they all go towards making Hadoop more Enterprise ready. Hadoop is already ready for the Enterprise when it comes to Scalability, Availability and Reliability. At Yahoo! we know this probably better than anyone else. But it is the rest of these core “abilities” that will put Hadoop on the fast track to Enterprise deployment at companies that don’t have large Hadoop engineering teams in-house.

Before I wrap-up, here are a few little known facts about this year’s Summit that speak to all of the above:

1. The Summit had a 27% percent acceptance rate for submissions – it was very competitive and a great sign that the interest in Hadoop is driving high-quality technical content.
2. 50% of the ticket sales happened in the last two weeks. In other words, you all need to plan better so we can fit more of you. :-)
3. There were no sponsored tech talks this year, none at all – so if you’re looking for unbiased, useful technical content, the Summit tech talks were as pure as can be.
4. Multiple leading database vendors are currently evaluating Hadoop and HBase for internal use.
5. Hadoop is designed for use with non-reliable storage. ;-)

Looking forward to seeing the northern California Hadoopers at the July HUG. If you would like to attend, you can sign up here: http://www.meetup.com/hadoop/.

These are exciting times for Hadoop and may you enjoy living through them.

/later

Jay Rossiter, Senior Vice President of the Yahoo! Cloud Platform Group will open the show with a keynote around how Yahoo! is developing the next generation of Hadoop applications to handle big data, the important role that Hadoop plays in Yahoo!’s integrated technology ecosystem and how wide industry adoption of Hadoop is benefiting the entire community.

Also on the main stage, Facebook will discuss its use of Hadoop to power the Facebook Messages infrastructure and IBM will discuss how they used Hadoop to power supercomputer, Watson.

Additional conference highlights include some key sessions:

• Next Generation Apache Hadoop MapReduce: Arun Murthy, Yahoo!’s lead architect on the Hadoop Map-Reduce development team, will lead a discussion on the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution.
• Introducing HCatalog (Hadoop Table Manager): Alan Gates, Yahoo! architect for Pig and Howl, will provide an overview of HCatalog as well as the release/roadmap.
• Automated Rolling OS Upgrades for Yahoo! Hadoop Grids: Dan Romike, Yahoo! Hadoop Data and Grid Systems engineer, will detail how we are upgrading thousands of servers, the problems of system state management, and the operational workflows specific to a Hadoop grid environment.
• Case Studies of Hadoop Operations at Yahoo!: The Grid Operations team at Yahoo! operates about 40,000 servers running Hadoop in clusters of up to 4,200 servers. Charles Wimmer, Yahoo! senior service engineer of grid computing, will provide a dive deep into a series of case studies that exemplify these issues.
• Large Scale Math with Hadoop MapReduce: Nerd out and learn how Yahoo! established a new world record by computing the two quadrillionth bits of pi using Hadoop in July 2010. Widely covered in the news, the world record computation was composed of 35,000 MapReduce jobs, requiring 23 days of real time and 503 years of CPU time in Yahoo! clusters. Led by Yahoo! Hadoop engineer Tsz-Wo Sze, attendees will also learn MapReduce algorithms for large-scale mathematical calculation, their implementation, and our experience in running and tuning these computations in Hadoop clusters.

Check out the official conference agenda for a full preview of what’s to come and a look at the 32 different sessions, including best practice deep-dives and case studies on the Hadoop roadmap, operations and management, innovative Hadoop applications and research, and much more.

Space is limited, so don't miss this unique opportunity to hear directly from Hadoop thought leaders and pioneers by registering now.

And finally, a special thanks to the Hadoop Summit 2011 sponsors:

Platinum Sponsors:
* MapR Technologies
* NetApp

Gold Sponsors:
* Aster Data
* Cloudera
* DataStax
* Datameer
* IBM

Silver Sponsors:
* Amazon Web Services
* Arista
* Impetus
* Pentaho
* Syncsort

Sponsors:
* Dell
* Hadapt
* HStreaming
* Jive
* Karmasphere
* Mellanox
* Pervasive DataRush
* Quest Software
* Softlayer
* StackIQ
* ThinkBig Analytics

Stay up to date on Hadoop Summit buzz by following #hadoopsummit on Twitter.

If you are a Hadooper or would like to become one, come join the community for this one day event.

Here are my slides from the IBM big data symposium. This was a good event. IBM announced a new release of their Apache Hadoop based Big Insights platform. It is great to hear their commitment to Apache. Yahoo was there talking about our experiences and uses of Hadoop. I got a lot of questions about why we invest in Hadoop, so let me point you back to my post on that and our commitment to Apache Hadoop. (http://yhoo.it/e8p3Dd and http://yhoo.it/i9Ww8W)

Thanks, E14

http://developer.yahoo.com/events/hadoopsummit2011/

The deadline is 6 May, which is tomorrow!

This environment is especially great if you have a great idea that you would like to share with the community. You will have a great audience of knowledgeable developers that you can try to convince to help you to take your work to the next level. Doesn't it sound ... great!?!

Milind and I are organizing the research and application track. If you have built some new framework on top of Hadoop or made Hadoop better, let us know. We will be selecting the most interesting results for the research and application track.

General information for the Hadoop Summit is at http://hadoopsummit.org. You can submit an abstract for your presentation at http://developer.yahoo.com/events/hadoopsummit2011/presentationguidelines.html

We are looking for presentations of new applications or improvements that have been implemented. We are not looking for position papers or visions. It is fine to submit a presentation that has been presented in other venues. If you are submitting something that has been published in a research conference, you may want to put an engineering perspective on it. This crowd looks behind the beautiful ideas and is going to want to see how/if it really works.

Hope to see you there!

Why Did We Create HCatalog?

Out of the box Hadoop provides the HDFS file system for users to store their data. File systems are nice because they provide a simple interface. Users can easily copy data into the file system and run jobs against that data. However, for more complex data processing tasks, the file system abstraction is not rich enough. It forces users to know where data is located, what format it is stored in, how it is compressed, and what its schema is. Consider, for example, a Pig Latin script used to do ETL on raw web logs:

A = load '/data/raw/ds=20110225/region=us/property=news' using PigStorage()
      as (user:chararray, url:chararray, timestamp:long);
...

If the grid administrator needs to move this data to a new location, or compact multiple files into an archive (such as har), or the data producer changes the schema or starts writing it compressed, this Pig Latin script will need to be changed. Pig and MapReduce job specifications are tightly coupled with the data storage. This inhibits data producers' ability to improve their processes. They and their users are forced to go through a painful transition process to take advantage of any improvements. This lack of data storage independence is exacerbated by the fact that in most large Hadoop installations there will be users using different tools to access their data. So a change in storage format will ripple through multiple groups and require coordination across disparate data processing tools.

The fact that in many Hadoop installations different users will be using different tools brings up another issue. These tools do not share a common notion of schemas or data types. Pig and Hive have different, though similar, data models. MapReduce leaves data types to its users. This can make sharing a data set across users with different tools difficult and error prone.

When data consumers are building new applications or want to query new data sets, finding the data in a file system is difficult. Generally, a user has to contact the group that produces the data and ask them where they write their data, what format it is in, and what its schema is. Some organizations use wiki pages or similar mechanisms to record this information. But such pages inevitably get stale as changes to the data storage are made.

Even once users know where data is, file systems do not help them know when it is available. In a complex data processing environment there will be many users waiting for the availability of a given data set. This is particularly true of the foundational data sets that form the basis for much of an organization's data processing. With a file system, the only way to know if data is available is to do an 'ls'. Hundreds of users banging away with 'ls' commands every few seconds does not help HDFS' performance.

Finally, a file system tends to become a dumping ground with little knowledge of how the data there should be managed. How long should data be kept? Are there legal reasons to store it on tape before deleting it? If so, how do you know what has been archived and what has not? With a file system all of these issues tend to be addressed by conventions. "Data in this directory is cleaned up after 30 days." "After data is archived a '.archive' file is placed in its directory." Often each data creator has to answer these questions, and they tend to answer them differently. Thus an organization ends up with not one data management policy, but twenty, thirty, or a hundred policies.

How HCatalog Addresses These Issues

Hadoop needs a better abstraction for data storage, and it needs a metadata service. HCatalog addresses both of these issues. It presents users with a table abstraction. This frees them from knowing where or how their data is stored. It allows data producers to change how they write data while still supporting existing data in the old format so that data consumers do not have to change their processes. It provides a shared schema and data model for Pig, Hive, and MapReduce. It will enable notifications of data availability. And it will provide a place to store state information about the data so that data cleaning and archiving tools can know which data sets are eligible for their services.

HCatalog takes Hive's metastore, and wraps additional layers around it to provide these services. It comes with HCatInputFormat and HCatOutputFormat for MapReduce users, and HCatLoader and HCatStorer for Pig users. Taking the Pig script example above, using HCatalog it looks like:

A = load 'raw' using HCatLoader();
B = filter A by ds='20110225' and region='us' and property='news';
...

Notice that Pig no longer knows or cares about the file location or storage format. It is just loading a table named 'raw'. If an administrator decides to relocate the files that store raw, or switch to a better storage format, this Pig Latin script does not change at all. Also notice that the user no longer has to declare the schema. HCatLoader communicates that automatically to Pig. HCatalog uses Hive's data model. When loading data into Pig, the types are translated to Pig types. For example, a Hive struct becomes a Pig tuple. When used by MapReduce, the value that HCatInputFormat returns is an HCatRecord, which is an ordered list of typed data. At the same time, this does not force Pig or MapReduce to scan the whole table. Pig knows how to push the filter statement into HCatLoader so that only the appropriate files are read.

HCatalog includes Hive's command line interface so that administrator can create and drop tables, specify table parameters, etc. It will also allow users to explore what tables are available and what their schema is.

HCatalog also provides an API for storage format developers to tell HCatalog how to read and write data stored in different formats. Currently HCatalog knows how to read and write RCFiles. But if data is stored in a different format, a user can implement an HCatInputStorageDriver and HCatOutputStorageDriver to tell HCatalog how to translate between your data storage and record format HCatalog uses. Which StorageDriver to use is stored at the partition level. So if you need to change how your table is stored when you already have a year's data in it, there is no need to reprocess the data. New data can be written in the new format while the old data stays in the old format. HCatalog handles using the correct StorageDriver to read each partition, allowing users to read across partitions, never knowing they are in different formats.

HCatalog Is Brought To You By...

HCatalog is a collaborative effort between members from the Apache Pig, Hive, and Hadoop projects, plus new contributors. Most of the new code has been written by Yahoos, while the Hive team has been very helpful in providing design feedback and pushing HCatalog's changes into the Hive metastore.

Where Can You Learn More?

You can learn more about HCatalog, download the source, file JIRAs, and join the mailing lists on HCatalog's website. You can also come join us at the Hadoop Summit.

After the usual 30 minutes of socializing and networking, Milind Bhandarkar from LinkedIn, kicked off the evening with a really enlightening talk on "Scaling Hadoop Applications." As a well-respected Hadoop expert and a founding member of the Hadoop team at Yahoo in 2005, Milind was able to articulate the issues and solutions very succinctly. His talk was especially interesting because he tied well known theorems and laws around scalability to the ground realities on the Hadoop clusters today.

Here are the slides from Milind's talk.

Following is the video of the presentation.

This was followed by an interesting talk on "HDFS Federation" by Yahoo's Suresh Srinivas. HDFS Federation is a major feature slated to come out in the near future and Suresh gave the audience an in-depth and in-the-trenches look at this key feature. This also tied into the theme of the day nicely as federation is all about scaling today's Hadoop clusters and making them bigger and faster.

You can find the slide-deck from Suresh's presentation below.

This talk concluded an interesting HUG. Thanks to all the Hadoop users and presenters who attended the March HUG and hope to see you all again soon. If you have an interesting topic related to Hadoop and would like to present, please don't hesitate to contact us.

Calling all Hadoopers

Yahoo! is pleased to announce that this year’s Hadoop Summit is scheduled for June 29th at the Santa Clara Convention Center. Registration for the event is now open and offers an early bird special of $125, a savings of nearly 30% on the full ticket price of $175. This ends on May 1st, so register now to take advantage of this great offer.

Whether you are already running and managing a Hadoop installation, developing Hadoop-based applications or exploring how to adopt Apache Hadoop for your business, the summit provides a unique opportunity to gain deep insights into the world of Hadoop from the company that pioneered it. Learn about interesting and relevant real-world applications and find out about the latest Big Data research.

The summit brings together some of the most influential speakers in the Hadoop space. Our full agenda provides many informative tracks for developers, administrators, managers and researchers. A session focused on new and innovative applications using Hadoop which was ‘standing-room’ only last year will again be a highlight of the event. Stay tuned the agenda will be posted shortly as we continue to update it.

Last year, we had to close registration early as we reached capacity extremely quickly with over 1200 attendees. This year we expect to attract an even larger audience as Hadoop continues to go mainstream, so please make sure you register early because space is limited. Yahoo! is the world’s largest contributor to Apache Hadoop and is joined by a number of technology leaders like Amazon, China Mobile, Facebook, Linkedin and HP, all using Hadoop for their businesses. Come and find out how your company could benefit from using Hadoop.

To ensure you get a ticket, take advantage of our early bird special which costs $125, a savings of nearly 30% on the full ticket price of $175 – Register here. The Early Bird ticket price ends May 1st.

If you are working on projects on or using Hadoop, this is also a terrific opportunity for you to share your experience with thousands of fellow Hadoop users. To submit a session abstract for one of the tracks below you can find the presentation guidelines and detailed instructions here. Submission deadline is May 6th.

Based on community feedback and interest here are the tracks for this year:

Community: Presentations and discussions about Hadoop roadmap, contribution and best practices.

Operations and Management: Presentations and discussions about operations and management of Hadoop clusters and best practices.

Applications and Research: Case studies about innovative applications and Hadoop technologies based research studies.

We’re excited to be bringing you the Hadoop Summit 2011, and look forward to seeing you there.

A groundbreaking open source project has won the top prize at the 2011 MediaGuardian Innovation Awards.

The judging panel described the Apache Hadoop project as the Swiss army knife of the 21st Century, and having the potential to completely change the face of media innovations across the globe. Overall, the project was seen as a greater catalyst for innovation than WikiLeaks, the iPad and a host of other suggested nominees.

All of the Hadoop contributors should be very proud of this award. Sanjay Radia, Jakob Homan, and I attended in person as members of the Hadoop Project Management Committee to receive the award on behalf of the project.

I've been working on Hadoop full time since the beginning and it has been a pleasure working with such bright and dedicated engineers. It takes a village to raise an elephant from a prototype that runs on a few nodes to the project that is disrupting the big data industry.

Congratulations Hadoop team!

Update: Here is a picture from the Guardian:

Iain Lee presenting the award to Sanjay, Jakob, and Owen on behalf of the project.

The previous post in this series covered the next generation of Apache Hadoop MapReduce in a broad sense, particularly its motivation, high-level architecture, goals, requirements, and aspects of its implementation.

In the second post in a series unpacking details of the implementation, we’d like to present the protocol for resource allocation and scheduling that drives application execution on a Next Generation Apache Hadoop MapReduce cluster.

Background

Apache Hadoop must scale reliably and transparently to handle the load of a modern, production cluster on commodity hardware. One of the most painful bottlenecks in the MapReduce framework has been the JobTracker, the daemon responsible not only for tracking and managing machine resources across the cluster, but also for enforcing the execution semantics for all the queued and running MapReduce jobs. The fundamental shift we hope to effect takes these two complex and interrelated concepts and re-factors them into separate entities.

In the proposed new architecture a global ResourceManager (RM) tracks machine availability and scheduling invariants while a per-application ApplicationMaster (AM) runs inside the cluster and tracks the program semantics for a given job. An application is either a single MapReduce job as the JobTracker supports today, it could be a directed, acyclic graph (DAG) of MapReduce jobs, or it could be a new framework. Each machine in the cluster runs a per-node daemon, the NodeManager (NM), responsible for enforcing and reporting the resource allocations made by the RM and monitoring the lifecycle of processes spawned on behalf of an application. Each process started by the NM is conceptually a container, or a bundle of resources allocated by the RM.

Scheduler

The ResourceManager is the central authority that arbitrates resources among various competing applications in the cluster.

The ResourceManager has two main components:

• Scheduler
• ApplicationsManager

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed containers either due to application failure or hardware failures.

The Scheduler performs its scheduling function based the resource requirements of the applications. Applications express their resource requirements using the abstract notion of resource request, which incorporates elements such as memory, cpu, disk, network etc.

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application-specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.

This post focuses on providing details about the Scheduler, the resource-model, the scheduling algorithm etc.

Resource Model

MapReduce NextGen supports a very generic resource model that allows cluster resources such as CPU, memory, disk and network bandwidth, local-storage etc. to be efficiently scheduled among applications.

The result of a resource request is a container. A container is a conceptual entity that grants an application the privilege to use a certain amount of resources on a given machine to run a component task.

The Scheduler, in v1, models every machine in the system as composed of multiple containers (with a lower-limit on the size of memory, say 512MB or 1 GB).

The ApplicationMaster request a container along with the resources (amount of memory, cpu capability, etc) it needs. It can request multiple type of containers, each with its different resource needs. In addition it can optionally specify location preferences such as specific machine or rack; this allows for first class support of the MapReduce model of performing the computation close to the data being processed.

Thus, unlike current implementation of Hadoop Map-Reduce, the cluster is not artificially segregated into map and reduce slots. Every container is fungible, yielding huge benefits for cluster utilization. Currently, a well-known problem with Hadoop MapReduce is that jobs are bottle-necked on reduce slots and the lack of fungible resources is a severe limiting factor.

Resource Request

As described in the previous section, the ApplicationMaster can ask for resources with specific capabilities (memory, cpu etc.) on specific machines or racks. The ApplicationMaster can ask also ask for different kinds of resources for differing needs (e.g. maps and reduces have different resource necessities in the MapReduce paradigm). Among various resource requests, the ApplicationMaster can use request priorities to influence ordering of containers allocated by the Scheduler (e.g. run maps before reduces in MapReduce). All outstanding resource requests are serviced strictly in priority order, as defined by the application.

The ApplicationMaster can ask for any number of containers of a given request type (<priority, capability> ).

Thus, the resource request understood by the Scheduler is of the form:

<priority, (hostname/rackname/*), capability, #containers>

Resource Negotiation

The ApplicationMaster is responsible for computing the resource requirements of the application. In a MR application, the input-splits express its locality preferences. The MR ApplicationMaster translates them into resource requests for the Scheduler.

The expected mode of operation is for applications to present all of their resource requests upfront, on start-up, to the extent feasible. Of course, the protocol allows for applications to present updates to the resource requests via deltas.

As described above the resource request is of the form: <priority, (hostname/rackname/*), capability, #containers>.

Thus, for MapReduce applications, the MapReduce ApplicationMaster takes the input-splits and generates resource requests by inverting the split to host mappings, then annotating each request with priorities (map v/s reduce containers) and number of containers required on each host. The resource requests are also aggregated by racks and then by the special any (*) for all containers. All resource requests are subject to change via the delta protocol.

An illustrative example:

Consider a MapReduce job with 2 maps, each requiring 1G of RAM and 1 reduce requiring 2G of RAM.

The input-splits are:

The corresponding resource requests from the ApplicationMaster to the Scheduler:

Notice how the containers field is aggregated across the different hosts, racks and any (*) for both maps and reduces.

The main advantage of the proposed model is that it is extremely compact in terms of the amount of state necessary, per application, on the ResourceManager for scheduling and the amount of information passed around between the ApplicationMaster & ResourceManager. This is crucial for scaling the ResourceManager. The amount of information, per application, in this model is always O(cluster size), whereas in the current Hadoop Map-Reduce JobTracker it is O(number of tasks) which could run into hundreds of thousands of tasks. For large jobs it is sufficient to ask for containers only on racks and not specific hosts since the ApplicationMaster can use them appropriately since each rack has many appropriate resources (i.e. input splits for MapReduce applications).

Scheduling

The Scheduler will try to match appropriate machines for the application; it can also provide resources on the same rack or a different rack if the specific machine is unavailable. The ApplicationMaster might, at times, due to vagaries of busy-ness of the cluster, receive resources that are not the most appropriate or receive them too late; it can then reject them by returning them to the Scheduler without being charged for them.

The Scheduler is aware of the network topology of the cluster and uses the racks and any (*) fields as gating factors to ensure that ApplicationMasters do not get inappropriate resources without resorting to needing very detailed task-level information.

For e.g. in the above example the Scheduler doesn’t need to know that map0 can run on <r11/h1001, r22/h2121, r31/h3118>.

If a container is available on r22/h2121, the Scheduler allocates it to the application and invalidates requests on h2121, r22 and * by updating the resource request list as follows:

The ApplicationMaster can then, optionally, invalidate requests for h1001, h3118, r11 and r31 after it decides to run map0 on h2121.

Now, if a container is available on r11/h1010, the Scheduler allocates it to the application and invalidates requests on h1010, r11 and * as follows:

The ApplicationMaster can then, optionally, invalidate requests for h2121, h4123, r22 and r45 after it decides to run map1 on h1010.

The important detail about the scheduling policy is that:

• When the number of required containers reaches 0 on a host, the Scheduler will stop granting further containers on that node to a the application at that priority/capability.
• When the number of required containers reaches 0 on a rack, the Scheduler will stop granting further containers to any node on that rack to the application at that priority/capability.
• When * reaches 0 for an application, the Scheduler stops granting further containers on any node for the application at that priority/capability.

The Scheduler aggregates resource requests from all the running applications to build a global plan to allocate resources. The Scheduler then allocates resources based on application-specific constraints such as appropriate machines and global constraints such as capacities of the application, queue, user etc.

The CapacityScheduler plug-in uses familiar notions of Capacity and Capacity Guarantees as the policy for arbitrating resources among competing applications.

The scheduling algorithm is straightforward:

• Pick the most under-served queue in the system.
• Pick the highest priority application in that queue.
• Serve the application's resource asks appropriately.

The FairScheduler plug-in uses fairness the main policy for arbitrating resources among active applications.

Note: A rogue ApplicationMaster could, in any scenario, request more containers than necessary - the Scheduler uses application-limits, user-limits, queue-limits etc. to protect the cluster from abuse.

Scheduler API

There is a single API between the Scheduler and the ApplicationMaster:

(List <Container> newContainers, List <ContainerStatus> containerStatuses) allocate (List <ResourceRequest> ask, List<Container> release)

The AM ask for specific resources via a list of ResourceRequests (ask) and releases unnecessary Containers which were allocated by the Scheduler.

The response contains a list of newly allocated Containers and the statuses of application-specific Containers that completed since the previous interaction between the AM and the RM.

Resource Monitoring

The Scheduler receives periodic information about the resource usages on allocated resources from the NodeManagers. The Scheduler also makes available status of completed Containers to the appropriate ApplicationMaster.

In future, obvious enhancements to the Scheduler include using the per-container resource usage information provided by the NodeManagers to do over-committing of resources etc.

Conclusions

We believe the proposed ideas for the Scheduler in the Next Generation of Hadoop Map-Reduce are both novel and crucial for the fine-grained scheduling required to meet our stated goals for Apache Hadoop MapReduce. They allow the Scheduler to scale to very large clusters and still provide very fine-grained and efficient scheduling for exploiting data-locality, performance and reliability.

We expect and encourage other cluster schedulers/managers to incorporate our ideas and we look forward to learning from their experiences too.