<p>By Mohit Goenka, Senior Engineering Manager</p><p><a href="https://yahooeng.tumblr.com/post/162320493306/yahoo-mails-new-tech-stack-built-for-performance">Building the technology</a> powering the best consumer email inbox in the world is no easy task. When you start on such a journey, it is important to consider how to deliver such an experience to the users. After all, any consumer feature we build can only make a difference after it is delivered to everyone via the tech pipeline. </p><p>As we began building out the <a href="https://yahoomail.tumblr.com/post/162320728976/a-cleaner-faster-and-more-powerful-yahoo-mail">new version of Yahoo Mail</a>, we wanted to ensure that our internal developer productivity would not be hindered by how our pipelines work. Keeping this in mind, we identified the following principles as most important while designing the delivery pipeline for the new Yahoo Mail experience: </p><ul><li>Product updates are pushed at regular intervals</li><li>Releases are stable</li><li>Builds are not blocked by irrational test failures</li><li>Developers are notified of code pushes</li><li>Hotfixes</li><li>Rollbacks</li><li>Heartbeat pushes </li></ul><p><b>Product updates are pushed at regular intervals</b> </p><p>We ensure that our engineers can push any code changes to all Mail users everyday, with the ability to push multiple times a day, if necessary or desired. This is possible because of the time we spent building a solid testing infrastructure, which continues to evolve as we scale to new users and add new features to the product. Every one of our builds runs 10,000+ unit tests and 5,000+ integration tests on various combinations of operating systems and browsers. It is important to push product updates regularly as it allows all our users to get the best Mail experience possible. </p><p><b>Releases are stable</b> </p><p>Every code release starts with the company’s internal audience first, where all our employees get to try out the latest changes before they go out to production. This begins with our <i>alpha</i> and <i>beta</i> environments that our Mail engineers use by default. Our build then goes out to the <i>canary</i> environment, which is a small subset of production users, before making it to all users. This gives us the ability to analyze quality metrics on internal and <i>canary</i> servers before rolling the build out to 100% of users in production. Once we go through this process, the code pushed to all our users is thoroughly baked and tested. </p><p><b>Builds are not blocked by irrational test failures</b> </p><p>Running tests using web drivers on multiple browsers, as is standard when testing frontend code, comes with the problem of tests irrationally failing. As part the Yahoo Mail continuous delivery pipeline, we employ various novel strategies to recover from such failures. One such strategy is recording the data related to failed tests in the first pass of a build, and then rerunning only the failed tests in the subsequent passes. This is achieved by creating a metadata file that stores all our build-related information. As part of this process, a new bundle is created with a new set of code changes. Once a bundle is created with build metadata information, the same build job can be rerun multiple times such that subsequent reruns would only run the failing tests. This significantly improves rerun times and eliminates the chances of build detentions introduced by irrational test failures. The recorded test information is analyzed independently to understand the pattern of failing tests. This helps us in improving the stability of those intermittently failing tests. </p><p><b>Developers are notified of code pushes</b> </p><p>Our build and deployment pipelines collect data related to all the authors contributing to any release through code commits or by merging various pull requests. This enables the build pipeline to send out email notifications to all our Mail developers as their code flows through each environment in our build pipeline (<i>alpha</i>, <i>beta</i>, <i>canary</i>, and <i>production</i>). With this ability, developers are well aware of where their code is in the pipeline and can test their changes as needed. </p><p><b>Hotfixes</b> </p><p>We have also created a pipeline to deploy major code fixes directly to production. This is needed even after the existence of tens of thousands of tests and multitudes of checks. Every now and then, a bug may make its way into production. For such instances, we have <i>hotfixes</i> that are very useful. These are code patches that we quickly deploy on top of production code to address critical issues impacting large sets of users. </p><p><b>Rollbacks</b> </p><p>If we find any issues in production, we do our best to minimize the impact on users by swiftly utilizing rollbacks, ensuring there is zero to minimal impact time. In order to do rollbacks, we maintain lists of all the versions pushed to production along with their release bundles and change logs. If needed, we pick the stable version that was previously pushed to production and deploy it directly on all the machines running our production instance. </p><p><b>Heartbeat pushes</b></p><p>As part of our continuous delivery efforts, we have also developed a concept we call <i>heartbeat</i> pushes. Heartbeat pushes are notifications we send users to refresh their browsers when we issue important builds that they should immediately adopt. These can include bug fixes, product updates, or new features. Heartbeat allows us to dynamically update the latest version of Yahoo Mail when we see that a user’s current version needs to be updated.</p><p><a href="https://s.yimg.com/ge/default/691231/YahooMailCICDFlow.jpg"></a></p><p><a href="https://s.yimg.com/ge/default/691231/YahooMailCICDFlow.jpg"></a></p><a href="https://s.yimg.com/ge/default/691231/YahooMailCICDFlow.jpg"><figure class="tmblr-full" data-orig-height="444" data-orig-width="1247" data-orig-src="https://s.yimg.com/ge/default/691231/YahooMailCICDFlow.jpg"><img src="https://66.media.tumblr.com/67782b9c91abd5a06750af5857a01e8c/tumblr_inline_pek5dnabG51rgj0aw_540.jpg" alt="image" data-orig-height="444" data-orig-width="1247" data-orig-src="https://s.yimg.com/ge/default/691231/YahooMailCICDFlow.jpg"/></figure></a><center><address dir="ltr">Yahoo Mail Continuous Delivery Flow</address></center><br/><p></p><p>In building the new Yahoo Mail experience, we knew that we needed to revamp from the ground up, starting with our continuous integration and delivery pipeline. The guiding principles of our new, forward-thinking infrastructure allow us to deliver new features and code fixes at a very high launch velocity and ensure that our users are always getting the latest and greatest Yahoo Mail experience.</p>