• Get Started
• Home
• About Us
• Developer Network Blog
• YDN Theater
• Frequently Asked Questions
• Support Communities
• Got Suggestions?
• Developer Central
• Java Developer Center
• JavaScript Developer Center
• Flash Developer Center
• ColdFusion Developer Center
• .NET Developer Center
• PHP Developer Center
• Python Developer Center
• Ruby Developer Center
• Silverlight Developer Center
• Lessons and Libraries
• Applications Gallery
• Browser Based Authentication
• Design Patterns
• Exceptional Performance
• Security Best Practices
• YUI - Yahoo! User Interface
• Work With Us
• Get an Application ID
• Merchant Solutions
• Partnerships
• Search Marketing
• Usage Policy
• Terms of Use
• Contact Us
• Services
• Answers
• Local
• Mail
• Maps
• OpenID
• Search
• Shopping
• Travel
• Utilities
• Sites
• del.icio.us
• Fire Eagle™
• Flickr™
• Live
• MyBlogLog
• Pipes
• Upcoming
• Yahoo! Research Berkeley
• Feeds
• Finance
• HotJobs
• RSS Feeds
• Traffic
• Weather
• Developer Kits
• Media Player
• Messenger
• Mobile
• Music Engine Plugins
• r3
• Search Developer Kit
• Widgets

« Announcing the Hadoop Summit at Yahoo, March 25th, 2008 | Main | Upcoming HBase User Group Meeting In San Francisco »

Mailtrust Hadoop Talk in Virginia on Monday

February 22, 2008

Just a quick heads up to Hadoop fans in the Virginia area. Bill Boebel, CTO of Mailtrust, will be giving a MapReduce vs. SQL Talk on Monday the 25th. (Mailtrust is the email division of Rackspace, a large hosting provider.)

Stu Hood, one of Mailtrust's software engineers wrote about MapReduce at Rackspace back in January, detailing how they use Hadoop for processing "several hundred gigabytes of email log data" every day.

The way it works is that raw logs get streamed from hundreds of mail servers to the Hadoop Distributed File System (”HDFS”) in real time, and scheduled MapReduce jobs run to index the new data using Apache Lucene and Solr. Once the indexes have been built, they are compressed and stored away in HDFS. Each Hadoop datanode also runs a Tomcat servlet container, which hosts a number of Solr instances that pull and merge the new indexes, and provide really fast search results to our support team.

Additionally, using MapReduce we are now able to look at our log data in all sorts of interesting ways. For example, we run nightly MapReduce jobs to collect statistics about our mail system, such as spam counts by domain, bytes transferred and number of logins. Now whenever we think of complex question about our customers’ usage patterns, we can pull the answer from our logs within hours via MapReduce. This is powerful stuff.

Read the whole posting for some interesting email stats they extracted.

Bill's talk should provide an excellent overview of Hadoop and some good insight into the Rackspace deployment.

Jeremy Zawodny
Yahoo! Developer Network

Posted at February 22, 2008 11:11 AM

   

Comments

Posted by: Trung Nguyen at February 22, 2008 5:59 PM

Post a comment

Name:

Email Address:

URL:

Remember Me? YesNo

Type "yahoo" here:

Comments:

Hadoop is a trademark of the Apache Software Foundation.