« Getting Paid to Test Open Source Software | Main | Hadoop is an Apache top level project (TLP) »

If it hurts, automate it

December 22, 2007

In many projects, it's painful to ensure that all unit tests are run on every patch before committing. Add to this some other basic checks:

no new javac compiler warnings
no new Findbugs warnings
zero javadoc warnings
zero @author attributions

and committing patches becomes very painful and time consuming indeed.

Something that painful, of course, is begging to be automated -- and that's exactly what we did with our patching process for the Hadoop project.

Every patch for Hadoop must be attached to a Jira issue. When a Jira issue is moved into the Patch Available state by a developer, the Hadoop continuous integration server automatically picks up the issue's latest patch, applies it to a fresh checkout of trunk, builds the software, runs all the unit tests, and verifies all the other items listed above. Once the build and testing is complete, a comment like this is automatically added to the Jira issue:

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368715/1185_20071030b.patch
against trunk revision r590273.

  @author +1. The patch does not contain any @author tags.
  javadoc +1. The javadoc tool did not generate any warning messages.
  javac +1. The applied patch does not generate any new compiler warnings.
  findbugs +1. The patch does not introduce any new Findbugs warnings.
  core tests -1. The patch failed core unit tests.
  contrib tests -1. The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/console

This message is automatically generated.

This automated system diminishes a Hadoop committer's burden to simply ensuring these two comments exist on the Jira issue:

a "+1" comment from the automated patch build, and
a "+1" comment from a code reviewer.

No more committing pain!

Nigel Daley
Grid Computing QA Lead

Want a fun job testing open source? Passionate about software quality? I'm hiring! Talk to me at: ndaley at yahoo-inc dot com. Testing and coding experience required.

Posted at December 22, 2007 8:22 AM

Comments

I was hoping you could elaborate on how you set up both Jira and the Hudson job to manage this.

Posted by: Ned Collyer at February 4, 2008 3:59 PM

Hi Ned,

All the Jira state changes are sent to the Hadoop mailing list. A daemon on the build machine is subscribed to the mailing list and processes the mail using this script. The patch "queue" created by that script is administered by the Hadoop-Patch-Admin build which runs this script every few minutes. When there is a patch to test, the Hadoop-Patch build is kicked off running this script.

I use a command line tool, jira_cli, to access Jira via it's SOAP interface.

Posted by: Nigel Daley at February 6, 2008 2:52 PM

If it hurts, automate it (Hadoop and Distributed Computing at Yahoo!)

If it hurts, automate it

Comments

Post a comment

Comments: