If it hurts, automate it

In many projects, it's painful to ensure that all unit tests are run on every patch before committing. Add to this some other basic checks:

  • no new javac compiler warnings
  • no new Findbugs warnings
  • zero javadoc warnings
  • zero @author attributions

and committing patches becomes very painful and time consuming indeed.

Something that painful, of course, is begging to be automated -- and that's exactly what we did with our patching process for the Hadoop project.

Every patch for Hadoop must be attached to a Jira issue. When a Jira issue is moved into the Patch Available state by a developer, the Hadoop continuous integration server automatically picks up the issue's latest patch, applies it to a fresh checkout of trunk, builds the software, runs all the unit tests, and verifies all the other items listed above. Once the build and testing is complete, a comment like this is automatically added to the Jira issue:

-1 overall. Here are the results of testing the latest attachment


against trunk revision r590273.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/console
This message is automatically generated.

This automated system diminishes a Hadoop committer's burden to simply ensuring these two comments exist on the Jira issue:

  • a "+1" comment from the automated patch build, and
  • a "+1" comment from a code reviewer.

No more committing pain!

Nigel Daley
Grid Computing QA Lead

Want a fun job testing open source? Passionate about software quality? I'm hiring! Talk to me at: ndaley at yahoo-inc dot com. Testing and coding experience required.