• Syntax cleanup of Hadoop fs shell commands
  The syntax of many Hadoop fs shell commands has been revised. The goals have syntax consistency across commands and compatibility with POSIX syntax as far as possible. For examples, see HADOOP-1677, HADOOP-1792, HADOOP-1891
• Block placement changes
  HDFS’ block placement strategy is now more tuned towards evenly distributing data across nodes. When a block is written from a node that is not running a Datanode, the first replica is written to a node on the same switch as the writer (instead of being written to the same node as the writer). The remaining two replicas are written to two nodes on a different switch. This strategy produces substantially better distributions in cases where data is loaded into HDFS from a small number of machines. More details at HADOOP-2559.
• Append... getting closer
  Everyone’s favorite bug, HADOOP-1700, is still not closed, but a lot of progress has been made in this release. Append related work will make possible new flush() and sync() methods in the HDFS interface. The semantics are familiar. The flush() call plonks data into the ‘system buffers’ and returns immediately. The sync() call writes the data to the system, and returns only when it has hit disk.
• More efficient replication
  Losing a rack of nodes usually means that the Namenode has to replicate around half a million blocks, and quickly. This would cause Namenode responsiveness to degrade and would increase the risk of data loss. A faster replication scheduling algorithm in the Namenode enables it to maintain quality of service to clients and replicate data faster in the event of losing many nodes at once.

Map/Reduce

• Switch awareness
  Hadoop map/reduce is now capable of switch aware task placement. The framework attempts to place tasks on machines where their input data resides. In many cases (in particular with HoD), machines with input data are not available for running tasks, but machines that share a switch with such ‘input nodes’ are. Hadoop will now attempt to place tasks on machines that are switch local to input when machines with input data are unavailable.
• Faster task scheduling
  HADOOP-2119 removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time.
• Sort and shuffle improvements
  A couple of significant improvements to sort and shuffle are included in the form of HADOOP-910 and HADOOP-2919. HADOOP-910 has reducers performing merges of shuffle data (both in memory and on disk) while fetching map outputs. HADOOP-2919 improves memory management in sort on the map side, substantially decreases setup cost for the sort and uses quicksort instead of mergesort as the sorting algorithm.

In 2006 we were building distributed applications that needed a master, aka coordinator, aka controller to manage the sub processes of the applications. It was a scenario that we had encountered before and something that we saw repeated over and over again inside and outside of Yahoo!.

For example, we have an application that consists of a bunch of processes. Each process needs be aware of other processes in the system. The processes need to know how requests are partitioned among the processes. They need to be aware of configuration changes and failures. Generally an application specific central control process manages these needs, but generally these control programs are specific to applications and thus represent a recurring development cost for each distributed application. Because each control program is rewritten it doesn't get the investment of development time to become truly robust, making it an unreliable single point of failure.

We developed ZooKeeper to be a generic coordination service that can be used in a variety of applications. The API consists of less than a dozen functions and mimics the familiar file system API. Because it is used by many applications we can spend time making robust and resilient to server failures. We also designed it to have good performance so that it can be used extensively by applications to do fine grained coordination.

We have found ZooKeeper to be applicable to many distributed applications inside of Yahoo! and expect it to be applicable to many more outside of Yahoo! For that reason we released it as open source under the Apache license. If you are writing a distributed application, ZooKeeper can help.

Powerset is hosting the first user group meeting for HBase, a robust, scalable, distributed, column-oriented store capable of hosting billions of row of sparse, structured data. HBase is open source and part of the Hadoop project.

Definitely come if you're a current user of HBase; or if your company has plans for a huge data store and you're evaluating solutions.

The way it works is that raw logs get streamed from hundreds of mail servers to the Hadoop Distributed File System (”HDFS”) in real time, and scheduled MapReduce jobs run to index the new data using Apache Lucene and Solr. Once the indexes have been built, they are compressed and stored away in HDFS. Each Hadoop datanode also runs a Tomcat servlet container, which hosts a number of Solr instances that pull and merge the new indexes, and provide really fast search results to our support team.

Additionally, using MapReduce we are now able to look at our log data in all sorts of interesting ways. For example, we run nightly MapReduce jobs to collect statistics about our mail system, such as spam counts by domain, bytes transferred and number of logins. Now whenever we think of complex question about our customers’ usage patterns, we can pull the answer from our logs within hours via MapReduce. This is powerful stuff.

Yahoo! is hosting the first summit on Apache Hadoop on March 25th in Sunnyvale. The summit is sponsored by the Computing Community Consortium (CCC) and brings together leaders from the Hadoop developer and user communities. Hadoop is now being used by companies in production environments, by academic and industrial research groups, and at universites for teaching data parallel computing. The speakers will cover topics in the areas of extensions being developed for Hadoop, case studies of applications being built and deployed on Hadoop, and a discussion on future directions for the platform.

• Number of links between pages in the index: roughly 1 trillion links
• Size of output: over 300 TB, compressed!
• Number of cores used to run a single Map-Reduce job: over 10,000
• Raw disk used in the production cluster: over 5 Petabytes

For example, in some cases we have an order of magnitude 10x improvement in indexing speed. This has been a multi-year project and we’re on track to have our future Search and advertising systems built on the new infrastructure, positioning us well for acceleration in iteration and experiments that are likely to lead to significant future product enhancements.

• Hadoop is now an Apache Top Level Project (Tom White)
• Hadoop - Top Level Project in Apache Software (jax magazine)

• no new javac compiler warnings
• no new Findbugs warnings
• zero javadoc warnings
• zero @author attributions

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368715/1185_20071030b.patch
against trunk revision r590273.

  @author +1. The patch does not contain any @author tags.
  javadoc +1. The javadoc tool did not generate any warning messages.
  javac +1. The applied patch does not generate any new compiler warnings.
  findbugs +1. The patch does not introduce any new Findbugs warnings.
  core tests -1. The patch failed core unit tests.
  contrib tests -1. The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/console

This message is automatically generated.

• a "+1" comment from the automated patch build, and
• a "+1" comment from a code reviewer.

• Hudson - open source continuous integration server
• Findbugs - static analyzer of Java code
• JUnit - unit test framework
• Checkstyle - coding style adherence tool
• Clover - commercial code coverage tool, donated to Apache
• Jira - commercial issue tracking tool, donated to Apache

The Incubator project is the entry path into The Apache Software Foundation (ASF) for projects and codebases wishing to become part of the Foundation's efforts. All code donations from external organisations and existing external projects wishing to join Apache enter through the Incubator.

Pig is a high-level language (PigLatin) for data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.