Build cache - Disk strategy
<h2>Build cache - Disk strategy</h2><p>Screwdriver now has the ability to cache and restore files and directories from your builds to either s3 or disk-based storage. Rest all features related to the cache feature remains the same, only a new storage option is added. Please DO NOT USE this cache feature to store any SENSITIVE data or information.</p><p>The graph below is our Internal Screwdriver instance build-cache comparison between disk-based strategy vs aws s3.</p><!-- more --><p>Build cache - get cache - (disk strategy)</p><figure data-orig-width="1182" data-orig-height="546" class="tmblr-full"><img src="https://66.media.tumblr.com/e21b7a705055194196e0ae630aab5218/83c06e8ef97325d7-e1/s540x810/1a98b177a22503dbc5068717052b1514b3167130.png" alt="image" data-orig-width="1182" data-orig-height="546"/></figure><p><b></b></p><p><b></b></p><p>Build cache - get cache - (s3)</p><figure data-orig-width="1186" data-orig-height="534" class="tmblr-full"><img src="https://66.media.tumblr.com/a2024316c758aa64e28877665701c927/83c06e8ef97325d7-ce/s540x810/44a03b2d35e8534baf87352158b3a627cdbc3339.png" alt="image" data-orig-width="1186" data-orig-height="534"/></figure><p>Build cache - set cache - (disk strategy)</p><figure data-orig-width="512" data-orig-height="225" class="tmblr-full"><img src="https://66.media.tumblr.com/5ac8de21ae69728e844dd80931a385c3/83c06e8ef97325d7-5c/s540x810/f88bc0fa78366a67d04def299eac56ed6acd19fb.png" alt="image" data-orig-width="512" data-orig-height="225"/></figure><p>Build cache - set cache - (s3)</p><figure data-orig-width="512" data-orig-height="232" class="tmblr-full"><img src="https://66.media.tumblr.com/60e4b7b20898df2835b519ce7cdc8ce7/83c06e8ef97325d7-9a/s540x810/0ff16bc281144f245cd42a79d0e39b85446016f9.png" alt="image" data-orig-width="512" data-orig-height="232"/></figure><p><br/></p><h2>Why disk-based strategy?</h2><p>Based on the cache analysis, 1. The majority of time was spent pushing data from build to s3, 2. At times the cache push fails if the cache size is big (ex: >1gb). So, simplified the storage part by using a disk cache strategy and using filer/storage mount as a disk option. Each cluster will have its own filer/storage disk mount.</p><p>NOTE: When a cluster becomes unavailable and if the requested cache is not available in the new cluster, the cache will be rebuilt once as part of the build.</p><h2><br/></h2><h2>Cache Size: </h2><p>Max size limit per cache is configurable by Cluster admins.</p><h2> </h2><h2>Retention policy:</h2><p>Cluster admins are responsible to enforce retention policy.</p><h2>Cluster Admins:</h2><p>Screwdriver cluster-admin has the ability to specify the cache storage strategy along with other options like compression, md5 check, cache max limit in MB</p><p>Reference: </p><ol><li><a href="https://github.com/screwdriver-cd/screwdriver/blob/master/config/default.yaml#L280">https://github.com/screwdriver-cd/screwdriver/blob/master/config/default.yaml#L280</a></li><li><a href="https://github.com/screwdriver-cd/executor-k8s-vm/blob/master/index.js#L336">https://github.com/screwdriver-cd/executor-k8s-vm/blob/master/index.js#L336</a></li><li>Issue: <a href="https://github.com/screwdriver-cd/screwdriver/issues/1830">https://github.com/screwdriver-cd/screwdriver/issues/1830</a></li></ol><p> <br/></p><h2>Compatibility List:</h2><p>In order to use this feature, you will need these minimum versions:</p><ul><li>API - <a href="https://github.com/screwdriver-cd/screwdriver/releases/tag/v0.5.835">v0.5.835</a></li><li>Buildcuster queue worker - v1.4.7</li><li>Launcher - <a href="https://github.com/screwdriver-cd/launcher/releases/tag/v6.0.42">v6.0.42</a></li><li>Store-cli - <a href="https://github.com/screwdriver-cd/store-cli/releases/tag/v0.0.50">v0.0.50</a></li><li>Store - <a href="https://github.com/screwdriver-cd/store/releases/tag/v3.10.3">v3.10.3</a></li></ul><h2><br/></h2><h2>Contributors:</h2><p>Thanks to the following people for making this feature possible:</p><ul><li><a href="https://github.com/parthasl">parthasl</a><br/></li></ul><p><br/></p><p>Screwdriver is an open-source build automation platform designed for Continuous Delivery. It is built (and used) by Yahoo. Don’t hesitate to reach out if you have questions or would like to contribute: <a href="https://t.umblr.com/redirect?z=http%3A%2F%2Fdocs.screwdriver.cd%2Fabout%2Fsupport.&t=MTdlNGVlY2FlNjQ1NTMwNWJhNzliNzA4MmQ3ZjAwMTJlYTYwYzZhZCxtd2o0dHBkaQ%3D%3D&b=t%3AjoJeie323MzP-suSfk7unw&p=https%3A%2F%2Fblog.screwdriver.cd%2Fpost%2F181683807942%2Fmultiple-build-cluster&m=1">http://docs.screwdriver.cd/about/support.</a></p>