Hadoop: The Future is Bright

hortonworks_200It’s been six years since the Hadoop project was started. These years have been an open source fairy tale. Starting from two developers with a vision, Hadoop now has a vibrant community of hundreds of contributors, and thousands of interested developers beyond that. Hadoop has gone from a powerful solution for specific problems to a foundational technology for almost any business. More than 1,600 developers from all walks of life will be at Hadoop Summit tomorrow. And there is a wealth of important work happening, not just on core Hadoop, but on the greater stack as well, with projects like Hbase, HIVE, PIG, and OOZIE making Hadoop ever more useful for a broad set of applications, from crunching Web search results and global retail transactions to analyzing genetic code.

From our early decision to financially support Hadoop and our subsequent huge engineering investment in it, to our community support with machines for build and test, the Summit, constant presence on mailing lists, support of authors writing Hadoop books, and our recent decision to revitalize the Apache distribution, Yahoo! has played a critical role in driving Hadoop's growth.

We constantly ask ourselves, “How can we take Hadoop to the next level?” While it’s been satisfying to see Hadoop gain so much momentum, we also understand that Hadoop will require even more investment and innovation to realize its full potential. In that spirit, Yahoo! and Benchmark Capital have announced the creation of Hortonworks a new, independent company focused on the standardization and commercialization of Hadoop, especially for the enterprise market. We’re confident this partnership will greatly increase the investment in core Hadoop, accelerate innovation, and further catalyze adoption.

As capital and commercial competition come into the ecosystem, it will be more challenging than ever to collaborate within the Hadoop Open Source community. With this in mind, here’s our plan:

1. Yahoo! will continue to invest in core Hadoop innovation and contribute our code to Apache Hadoop.
Hadoop powers major parts of Yahoo!’s business, from personalization technology on our home page to our global display advertising systems. We’re processing 5 billion jobs per month and have grown our Hadoop installation to more than 40,000 servers. We will continue to push the limits of what’s possible on Hadoop, and will continue passing along our knowledge and code to you through Apache.

2. Yahoo! will help Hortonworks support and contribute to the official Apache Hadoop project.
There’s an amazing opportunity to develop Hadoop solutions designed for the enterprise and for specific industries. Hortonworks is in a prime position to develop those solutions and expand the capabilities of Hadoop to a broader user base, which is why Yahoo! is thrilled to be lending its support. But Yahoo! has always believed that a strong, vibrant, and truly open community working on Hadoop is in the best interest of the whole industry. Hortonworks – which will be staffed by some of the strongest supporters of that philosophy at Yahoo! – will continue to operate on this core belief.

Hadoop has come a long way, and it’s been an amazing ride so far. No doubt Hadoop will continue to be one of the most fascinating software projects in the world.

I hope all of you attending Hadoop Summit tomorrow enjoy the event, get tons of insight out of the sessions, and keep growing your own Hadoop projects. Most importantly, I hope all of you will join with Yahoo! in continuing to support Apache Hadoop, so that it remains an open source fairy tale come true.

Raymie Stata, Chief Technology Officer - Yahoo!