Yahoo Developer Network

February 24, 2016

Share

CaffeOnSpark Open Sourced for Distributed Deep Learning on Big Data Clusters

By Andy Feng(@afeng76), Jun Shi and Mridul Jain (@mridul_jain), Yahoo Big ML Team <hr>Introduction Deep learning (DL) is a critical capability required by Yahoo product teams (ex. <a href="https://www.youtube.com/watch?v=uUvLxmsHpr4&list=PLnDbcXCpYZ8lCKExMs8k4PtIbani9ESX3&index=7" target="_blank">Flickr</a>, Image Search) to gain intelligence from massive amounts of online data. Many existing DL frameworks require a separated cluster for deep learning, and multiple programs have to be created for a typical machine learning pipeline (see Figure 1). The separated clusters require large datasets to be transferred among them, and introduce unwanted system complexity and latency for end-to-end learning.<figure data-orig-width="5427" data-orig-height="2124" class="tmblr-full"><img src="https://66.media.tumblr.com/b21c37c0fb9269699fbeb3c68d1b9242/tumblr_inline_o2vnzoXL1b1t17fny_540.jpg" alt="image" data-orig-width="5427" data-orig-height="2124"/></figure>Figure 1: ML Pipeline with multiple programs on separated clusters As discussed in our earlier <a href="http://yahoohadoop.tumblr.com/post/129872361846/large-scale-distributed-deep-learning-on-hadoop" target="_blank">Tumblr post</a>, we believe that deep learning should be conducted in the same cluster along with existing data processing pipelines to support feature engineering and traditional (non-deep) machine learning. We created CaffeOnSpark to allow deep learning training and testing to be embedded into Spark applications (see Figure 2). <figure data-orig-width="3166" data-orig-height="2114" class="tmblr-full"><img src="https://66.media.tumblr.com/d4fd5b7a14bd282e09f79b1c2af436e6/tumblr_inline_o2vo44cIg51t17fny_540.jpg" alt="image" data-orig-width="3166" data-orig-height="2114"/></figure>Figure 2: ML Pipeline with single program on one cluster CaffeOnSpark: API & Configuration and CLI CaffeOnSpark is designed to be a Spark deep learning package. <a href="http://spark.apache.org/docs/latest/mllib-guide.html" target="_blank">Spark MLlib</a> supported a variety of non-deep learning algorithms for classification, regression, clustering, recommendation, and so on. Deep learning is a key capacity that Spark MLlib lacks currently, and CaffeOnSpark is designed to fill that gap. <a href="http://yahoo.github.io/CaffeOnSpark" target="_blank">CaffeOnSpark API</a> supports <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html" target="_blank">dataframes</a> so that you can easily interface with a training dataset that was prepared using a Spark application, and extract the predictions from the model or features from intermediate layers for results and data analysis using MLLib or SQL.<figure data-orig-width="3385" data-orig-height="1916" class="tmblr-full"><img src="https://66.media.tumblr.com/ca004ff94918dca10d533cd118c8a914/tumblr_inline_o2vo8lJDh81t17fny_540.jpg" alt="image" data-orig-width="3385" data-orig-height="1916"/></figure>Figure 3: CaffeOnSpark as a Spark Deep Learning package <blockquote>1: def main(args: Array[String]): Unit = { 2: val ctx = new SparkContext(new SparkConf()) 3: val cos = new <a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.CaffeOnSpark" target="_blank">CaffeOnSpark</a>(ctx) 4: val conf = new <a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.Config" target="_blank">Config</a>(ctx, args).init() 5: val dl_train_source = <a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.DataSource%24" target="_blank">DataSource.getSource</a>(conf, true) 6: cos.<a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.CaffeOnSpark" target="_blank">train</a>(dl_train_source) 7: val lr_raw_source = <a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.DataSource%24" target="_blank">DataSource.getSource</a>(conf, false) 8: val extracted_df = cos.<a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.CaffeOnSpark" target="_blank">features</a>(lr_raw_source) 9: val lr_input_df = extracted_df.withColumn(“Label”, cos.floatarray2doubleUDF(extracted_df(conf.label)))10: .withColumn(“Feature”, cos.floatarray2doublevectorUDF(extracted_df(conf.features(0))))11: val lr = new LogisticRegression().setLabelCol(“Label”).setFeaturesCol(“Feature”)12: val lr_model = lr.fit(lr_input_df) 13: lr_model.write.overwrite().save(conf.outputPath) 14: }</blockquote>Figure 4: <a href="https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-grid/src/main/scala/com/yahoo/ml/caffe/examples/MyMLPipeline.scala" target="_blank">Scala application</a> using CaffeOnSpark both MLlib Scala program in Figure 4 illustrates how CaffeOnSpark and MLlib work together:<ul><li>L1-L4 … You initialize a Spark context, and use it to create CaffeOnSpark and configuration object. </li><li>L5-L6 … You use CaffeOnSpark to conduct DNN training with a training dataset on HDFS. </li><li>L7-L8 …. The learned DL model is applied to extract features from a feature dataset on HDFS. </li><li>L9-L12 … MLlib uses the extracted features to perform non-deep learning (more specifically logistic regression for classification). </li><li>L13 … You could save the classification model onto HDFS.</li></ul>As illustrated in Figure 4, CaffeOnSpark enables deep learning steps to be seamlessly embedded in Spark applications. It eliminates unwanted data movement in traditional solutions (as illustrated in Figure 1), and enables deep learning to be conducted on big-data clusters directly. Direct access to big-data and massive computation power are critical for DL to find meaningful insights in a timely manner.CaffeOnSpark uses the configuration files for solvers and neural network as in standard Caffe uses. As illustrated in our <a href="https://github.com/yahoo/CaffeOnSpark/blob/master/data/lenet_memory_train_test.prototxt" target="_blank">example</a>, the neural network will have a MemoryData layer with 2 extra parameters:<ol><li>source_class specifying a data source class </li><li>source specifying dataset location. </li></ol>The initial CaffeOnSpark release has several built-in data source classes (including <a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.LMDB" target="_blank">com.yahoo.ml.caffe.LMDB</a> for <a href="http://symas.com/mdb/doc/" target="_blank">LMDB</a> databases and <a href="http://yahoo.github.io/CaffeOnSpark/scala_doc/#com.yahoo.ml.caffe.SeqImageDataSource" target="_blank">com.yahoo.ml.caffe.SeqImageDataSource</a> for <a href="http://wiki.apache.org/hadoop/SequenceFile" target="_blank">Hadoop sequence files</a>). Users could easily introduce customized data source classes to interact with the existing data formats. CaffeOnSpark applications will be launched by standard Spark commands, such as spark-submit. Here are 2 examples of spark-submit commands. The first command uses CaffeOnSpark to train a DNN model saved onto HDFS. The second command is a custom Spark application that embedded CaffeOnSpark along with MLlib.First command: <blockquote>spark-submit \ –files caffenet_train_solver.prototxt,caffenet_train_net.prototxt \ –num-executors 2 \ –class com.yahoo.ml.caffe.CaffeOnSpark \ caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar \ -train -persistent \ -conf caffenet_train_solver.prototxt \ -model hdfs:///sample_images.model \ -devices 2 </blockquote>Second command: <blockquote>spark-submit \ –files caffenet_train_solver.prototxt,caffenet_train_net.prototxt \ –num-executors 2 \ –class com.yahoo.ml.caffe.examples.MyMLPipeline \ caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar \ -features fc8 \ -label label \ -conf caffenet_train_solver.prototxt \ -model hdfs:///sample_images.model \ -output hdfs:///image_classifier_model \ -devices 2</blockquote> System Architecture <figure data-orig-width="5572" data-orig-height="4135" class="tmblr-full"><img src="https://66.media.tumblr.com/7ccf1da404389778a4014340abd7dcb3/tumblr_inline_o2voaoqngI1t17fny_540.jpg" alt="image" data-orig-width="5572" data-orig-height="4135"/></figure>Figure 5: System Architecture Figure 5 describes the system architecture of CaffeOnSpark. We launch Caffe engines on GPU devices or CPU devices within the Spark executor, via invoking a JNI layer with fine-grain memory management. Unlike traditional Spark applications, CaffeOnSpark executors communicate to each other via MPI allreduce style interface via TCP/Ethernet or RDMA/Infiniband. This Spark+MPI architecture enables CaffeOnSpark to achieve similar performance as dedicated deep learning clusters.Many deep learning jobs are long running, and it is important to handle potential system failures. CaffeOnSpark enables training state being snapshotted periodically, and thus we could resume from previous state after a failure of a CaffeOnSpark job. Open Source In the last several quarters, Yahoo has applied CaffeOnSpark on several projects, and we have received much positive feedback from our internal users. Flickr teams, for example, made significant improvements on image recognition accuracy with CaffeOnSpark by training with millions of photos from the Yahoo Webscope <a href="http://webscope.sandbox.yahoo.com/catalog.php?datatype=i&did=67" target="_blank">Flickr Creative Commons 100M dataset</a> on Hadoop clusters. CaffeOnSpark is beneficial to deep learning community and the Spark community. In order to advance the fields of deep learning and artificial intelligence, Yahoo is happy to release CaffeOnSpark at <a href="https://github.com/yahoo/CaffeOnSpark" target="_blank">github.com/yahoo/CaffeOnSpark</a> under Apache 2.0 license. CaffeOnSpark can be tested on an <a href="https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_EC2" target="_blank">AWS EC2</a> cloud or on <a href="https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_local" target="_blank">your own Spark clusters</a>. Please find the detailed instructions at Yahoo <a href="https://github.com/yahoo/CaffeOnSpark" target="_blank">github</a> repository, and share your feedback at <a href="mailto:bigdata@yahoo-inc.com" target="_blank">bigdata@yahoo-inc.com</a>. Our goal is to make CaffeOnSpark widely available to deep learning scientists and researchers, and we welcome contributions from the community to make that happen. .

caffe spark hadoop ML machine learning apache open source

Latest Blogs

Latest Blogs

CaffeOnSpark Open Sourced for Distributed Deep Learning on Big Data Clusters