Grid Computing Archive

March 12, 2008

More Hadoop Summit Seats Available! New Venue too.

To say that we've been surprised by the interest in attending the Hadoop Summit would be an understatement. We already expanded the capacity once and that filled up in a matter of hours. And that pretty much maxed out the event budget and parking too.

So last week when our friends at Amazon Web Services got in touch to see if they could help, we started working on a plan to make the event even larger while still keeping it free. Before long, we'd hatched a plan that involved moving off-site to a nearby venue, more food, more T-shirts, and some minor schedule tweaking.

But most importantly, we have room for about 75 more people!

As of now, we've increased the capacity of the event on Upcoming.org. If you're been watching and waiting to get on the liste of attendees, the time is now.

The new venue is the Network Meeting Center which is located in Santa Clara, California.

Thanks to Amazon.com for buying everyone lunch during the summit. :-)

We'll be updating the agenda soon to include Jinesh from Amazon who will discuss GrepTheWeb - Hadoop on AWS. As you may know, Amazon has many customers running Hadoop on EC2.

If you cannot attend, we're still planning to record all the talks and put them on-line within a week after the summit date.

See you at the summit!

See Also: Hadoop Summit also scaled on-demand! on the Amazon Web Services blog.

Jeremy Zawodny
Yahoo! Developer Network

Posted by jzawodn at 10:57 AM | Comments (2) | TrackBack

March 7, 2008

An Introduction to ZooKeeper Video

A few weeks ago, I had the chance to capture video of a presentation given by Benjamin Reed from Yahoo! Research. His presentation was an introduction to ZooKeeper, a highly available and reliable coordination system built by Yahoo! Research and released under the Apache License, Version 2.0.

Preparing to post the video, I asked Ben for a a summary of the motivations for building ZooKeeper. Here's what he had to say:

In 2006 we were building distributed applications that needed a master, aka coordinator, aka controller to manage the sub processes of the applications. It was a scenario that we had encountered before and something that we saw repeated over and over again inside and outside of Yahoo!.

For example, we have an application that consists of a bunch of processes. Each process needs be aware of other processes in the system. The processes need to know how requests are partitioned among the processes. They need to be aware of configuration changes and failures. Generally an application specific central control process manages these needs, but generally these control programs are specific to applications and thus represent a recurring development cost for each distributed application. Because each control program is rewritten it doesn't get the investment of development time to become truly robust, making it an unreliable single point of failure.

We developed ZooKeeper to be a generic coordination service that can be used in a variety of applications. The API consists of less than a dozen functions and mimics the familiar file system API. Because it is used by many applications we can spend time making robust and resilient to server failures. We also designed it to have good performance so that it can be used extensively by applications to do fine grained coordination.

We have found ZooKeeper to be applicable to many distributed applications inside of Yahoo! and expect it to be applicable to many more outside of Yahoo! For that reason we released it as open source under the Apache license. If you are writing a distributed application, ZooKeeper can help.

And here's the video...

download (m4v)

A PDF copy of the slides is available too.

While filming his 1 hour presentation, I found myself really wishing that ZooKeeper was available 6 or 7 years ago when I was struggling with how to perform distributed processing of news feeds for Yahoo! Finance. ZooKeeper is clearly a more elegant solution than the hack we put together!

Ben will be speaking about ZooKeeper later this month at the Hadoop Summit.

More videos are available on YDN Theater.