<p><a class="tumblr_blog" href="http://yahooisrael.tumblr.com/post/153336432846" target="_blank">yahooisrael</a>:</p> <blockquote> <p><figure class="tmblr-full" data-orig-height="360" data-orig-width="540" data-orig-src="https://66.media.tumblr.com/42317c9a76e605b2368b35830c9c159f/tumblr_inline_ogt7giqQoR1rgj0aw_540.jpg"><img src="https://66.media.tumblr.com/1e3516b830ce3ee4526a04f7229eabbd/tumblr_inline_ogtvyotQv01rgj0aw_540.jpg" data-orig-height="360" data-orig-width="540" data-orig-src="https://66.media.tumblr.com/42317c9a76e605b2368b35830c9c159f/tumblr_inline_ogt7giqQoR1rgj0aw_540.jpg"/></figure></p> <p>By <a href="https://research.yahoo.com/researchers/ebortnik" target="_blank">Edward Bortnikov</a><b><br/></b></p> <p>The Apache Hadoop technology suite is the engine behind the Big Data revolution that has been transforming multiple industries over the last decade. Hadoop was born at Yahoo 10 years ago as a pioneering open-source project. It quickly outgrew the company’s boundaries to become a vehicle that powers thousands of businesses ranging from small enterprises to Web giants. </p> <p>These days, Yahoo is the largest Hadoop deployment in the industry. We run tens of thousands of Hadoop machines in our datacenters and manage more than 600 petabytes of data. Our products use Hadoop in a variety of ways that reflect a wealth of data processing patterns. Yahoo’s infrastructure harnesses Hadoop Distributed File System (HDFS) for ultra-scalable storage, Hadoop MapReduce for massive ad-hoc batch processing, Hive and Pig for database-style analytics, HBase for key-value storage, Storm for stream processing, and Zookeeper for reliable coordination. </p> <p>Yahoo’s commitment to Hadoop goes far beyond operating the technology at Web scale. The company’s engineers and scientists make contributions to both entrenched and incubating Hadoop projects. Our Scalable Platforms team at Yahoo Research in Haifa has championed multiple innovative efforts that have benefited Yahoo products as well as the entire Hadoop community. Just recently, we contributed new algorithms to HBase, <a href="http://yahoohadoop.tumblr.com/post/129089878751/introducing-omid-transaction-processing-for" target="_blank">Omid</a> (transaction processing system for HBase), and Zookeeper. Our work significantly boosted the performance of these systems and hardened their fault-tolerance. For example, the enhancements to Omid were instrumental for turning it into an <a href="http://incubator.apache.org/projects/omid.html" target="_blank">Apache Incubation project</a> (candidate for top-level technology status), whereas the work in HBase was named one of its top new features this year. </p> <p>Our team launched approximately three years ago. Collectively we add many years of experience to Yahoo and the Hadoop community in distributed computing research and development. We specialize in scalability and high availability, arguably the biggest challenges in big data platforms. We love to identify hard problems in large-scale systems, design algorithms to solve them, develop the code, experiment with it, and finally contribute to the community. The team features researchers with deep theoretical backgrounds as well as the engineering maturity required to deal with complex production code. Our researchers regularly present their innovations at leading industrial conferences (Hadoop Summit and HBaseCon), as well as at top academic venues. </p> <p>The researchers in our team comprise a blend of backgrounds in distributed computing, programming languages, and big systems, and most of us hold PhD degrees in these areas. We are especially proud to be a pioneering team of Hadoop developers in Israel. As such, we teach courses in big data technologies, organize technical meetups, and collaborate with academic colleagues. We are always happy to share our expertise with the ever-growing community of Hadoop users in the local hi-tech industry. </p> </blockquote> <p>Great contributions to Hadoop from our team in Haifa, Israel!</p>