<p><a href="https://www.linkedin.com/in/kishorkumarpatil/">Kishor Patil</a>, PMC Chair Apache Storm &amp; Sr. Principal Software Systems Engineer, Verizon Media <br/></p><p>Last year, we <a href="https://yahoodevelopers.tumblr.com/post/185258393008/apache-storm-20-improvements">shared with you</a> many of the Apache Storm 2.0 improvements contributed by Verizon Media. At Yahoo/Verizon Media, we’ve been committing to Storm for many years. Today, we’re excited to explore a few of the new features, improvements, and bug fixes we’ve contributed to Storm 2.2.0.<br/></p><p><b></b></p><p><b>NUMA Support </b></p><p><b></b></p><p>The server hardware is getting beefier and requires worker JVMs to be NUMA (Non-uniform memory access) aware. Without constraining JVMs to NUMA zones, we noticed dramatic degradation in the JVM performance; specifically for Storm where most of the JVM objects are short-lived and continuous GC cycles perform complete heap scan. This feature enables maximizing hardware utilization and consistent performance on asymmetric clusters. For more information please refer to [<a href="https://issues.apache.org/jira/browse/STORM-3259">STORM-3259</a>].</p><p><b></b></p><p><b>Auto Refreshing SSL Certificates for All Daemons</b></p><p>At Verizon Media, as part of maintaining thousands of Storm nodes, refreshing SSL/TLS certificates without any downtime is a priority. So we implemented auto refreshing SSL certificates for all daemons without outages. This becomes a very useful feature for operation teams to monitor and update certificates as part of hassle free continuous monitoring and maintenance. Included in the security related critical bug fixes the Verizon Media team noticed and fixed are:</p><ul><li>Kerberos connectivity from worker to Nimbus/Supervisor for RPC heartbeats [<a href="https://issues.apache.org/jira/browse/STORM-3579">STORM-3579</a>]</li><li>Worker token refresh causing authentication failure [<a href="https://issues.apache.org/jira/browse/STORM-3578">STORM-3578</a>]<br/></li><li>Use UserGroupInformation to login to HDFS only once per process [<a href="https://issues.apache.org/jira/browse/STORM-3494">STORM-3494</a>]<br/></li><li>AutoTGT shouldn&rsquo;t invoke TGT renewal thread [<a href="https://issues.apache.org/jira/browse/STORM-3606">STORM-3606</a>]<br/></li></ul><p><b></b></p><p><b>V2 Tick Backwards Compatibility</b></p><p><b></b></p><p>This allows for deprecated metrics at worker level to utilize messaging and capture V1 metrics. This is a stop-gap giving topology developers sufficient time to switch from V1 metrics to V2 metrics API. The Verizon Media Storm team also provided shortening metrics names to allow for metrics names that conform to more aggregation strategies by dimension [<a href="https://issues.apache.org/jira/browse/STORM-3627">STORM-3627</a>]. We’ve also started removing deprecated metrics API usage within storm-core and storm-client modules and adding new metrics at nimbus/supervisor daemon level to monitor activity.</p><p><b>Scheduler Improvements</b><br/></p><p><b></b></p><p>ConstraintSolverStrategy allows for max co-location count at the Component Level. This allows for better spread - [<a href="https://issues.apache.org/jira/browse/STORM-3585">STORM-3585</a>]. Both ResourceAwareScheduler and ConstraintSolverStrategy are refactored for faster performance. Now a large topology of 2500 component topology requesting complex constraints or resources can be scheduled in less than 30 seconds. This improvement helps lower downtime during topology relaunch - [<a href="https://issues.apache.org/jira/browse/STORM-3600">STORM-3600</a>]. Also, the blacklisting feature to detect supervisor daemon unavailability by nimbus is useful for failure detection in this release [<a href="https://issues.apache.org/jira/browse/STORM-3596">STORM-3596</a>].</p><p><b>OutputCollector Thread Safety</b><br/></p><p><b></b></p><p>For messaging infrastructure, data corruption can happen when components are multi-threaded because of non thread-safe serializers. The patch [<a href="https://issues.apache.org/jira/browse/STORM-3620">STORM-3620</a>] allows for Bolt implementations that use OutputCollector in other threads than executor to emit tuples. The limitation is batch size 1. This important implementation change allows for avoiding data corruption without any performance overhead. </p><p><b></b></p><p><b>Noteworthy Bug Fixes</b></p><p><b></b></p><ul><li>For LoadAwareShuffle Grouping, we were seeing a worker overloaded and tuples timing out with load aware shuffle enabled. The patch checks for low watermark limits before switching from Host local to Worker local - [<a href="https://issues.apache.org/jira/browse/STORM-3602">STORM-3602</a>].<br/></li><li>For Storm UI, the topology visualization related bugs are fixed so topology DAG can be viewed more easily.<br/></li><li>The bug fix to allow the administrator access to topology logs from UI and logviewer.<br/></li><li>storm cli bug fixes to accurately process command line options.<br/></li></ul><p><b><b>What’s Next</b><br/></b></p><p>In the next release, Verizon Media plans to contribute container support with Docker and RunC container managers. This should be a major boost with three important benefits - customization of system level dependencies for each topology with container images, better isolation of resources from other processes running on the bare metal, and allowing each topology to choose their worker OS and java version across the cluster. <br/></p><p><b>Contributors</b><br/></p><p><a href="https://www.linkedin.com/in/gresch">Aaron Gresch</a>, <a href="https://www.linkedin.com/in/meng-ethan-li-772bb1a7">Ethan Li</a>, <a href="https://www.linkedin.com/in/grmenon">Govind Menon</a>, <a href="https://www.linkedin.com/in/bipin-p-8477553">Bipin Prasad</a>, <a href="https://www.linkedin.com/in/rui-li-95855590">Rui Li</a></p>