<p><a href="https://yahoodevelopers.tumblr.com/post/181461308249/announcing-opentsdb-240-rollup-and" class="tumblr_blog">yahoodevelopers</a>:</p><blockquote> <p><i>By Chris Larsen, Architect</i><br/></p> <figure class="tmblr-full" data-orig-height="865" data-orig-width="4199"><img src="https://66.media.tumblr.com/8cb55742a24d71e55f9ae7530f93e87e/tumblr_inline_pkd9gcCWdK1w911gs_540.png" data-orig-height="865" data-orig-width="4199"/></figure><p><a href="http://opentsdb.net/">OpenTSDB</a> is one of the first dedicated open source time series databases built on top of Apache HBase and the Hadoop Distributed File System. Today, we are proud to share that version 2.4.0 is now available and has many new features developed in-house and with contributions from the open source community. This release would not have been possible without support from our monitoring team, the Hadoop and HBase developers, as well as contributors from other companies like Salesforce, Alibaba, JD.com, Arista and more. Thank you to everyone who contributed to this release!</p> <p>A few of the exciting new features include:</p> <p><b>Rollup and Pre-Aggregation Storage</b></p> <p>As time series data grows, storing the original measurements becomes expensive. Particularly in the case of monitoring workflows, users rarely care about last years’ high fidelity data. It’s more efficient to store lower resolution “rollups” for longer periods, discarding the original high-resolution data. OpenTSDB now supports storing and querying such data so that the raw data can expire from HBase or Bigtable, and the rollups can stick around longer. Querying for long time ranges will read from the lower resolution data, fetching fewer data points and speeding up queries.</p> <p>Likewise, when a user wants to query tens of thousands of time series grouped by, for example, data centers, the TSD will have to fetch and process a significant amount of data, making queries painfully slow. To improve query speed, pre-aggregated data can be stored and queried to fetch much less data at query time, while still retaining the raw data. We have an Apache Storm pipeline that computes these rollups and pre-aggregates, and we intend to open source that code in 2019. For more details, please visit <a href="http://opentsdb.net/docs/build/html/user_guide/rollups.html">http://opentsdb.net/docs/build/html/user_guide/rollups.html</a>.</p> <p><b>Histograms and Sketches</b></p> <p>When monitoring or performing data analysis, users often like to explore percentiles of their measurements, such as the 99.9th percentile of website request latency to detect issues and determine what consumers are experiencing. Popular metrics collection libraries will happily report percentiles for the data they collect. Yet while querying for the original percentile data for a single time series is useful, trying to query and combine the data from multiple series is mathematically incorrect, leading to errant observations and problems. For example, if you want the 99.9th percentile of latency in a particular region, you can’t just sum or recompute the 99.9th of the 99.9th percentile.</p> <p>To solve this issue, we needed a complex data structure that can be combined to calculate an accurate percentile. One such structure that has existed for a long time is the bucketed histogram, where measurements are sliced into value ranges and each range maintains a count of measurements that fall into that bucket. These buckets can be sized based on the required accuracy and the counts from multiple sources (sharing the same bucket ranges) combined to compute an accurate percentile.</p> <p>Bucketed histograms can be expensive to store for highly accurate data, as many buckets and counts are required. Additionally, many measurements don’t have to be perfectly accurate but they should be precise. Thus another class of algorithms could be used to approximate the data via sampling and provide highly precise data with a fixed interval. Data scientists at Yahoo (now part of Oath) implemented a great Java library called <a href="https://datasketches.github.io/">Data Sketches</a> that implements the Stochastic Streaming Algorithms to reduce the amount of data stored for high-throughput services. Sketches have been a huge help for the OLAP storage system <a href="http://druid.io/">Druid</a> (also sponsored by Oath) and <a href="https://bullet-db.github.io/">Bullet</a>, Oath’s open source real-time data query engine.</p> <p>The latest TSDB version supports bucketed histograms, Data Sketches, and <a href="https://github.com/OpenTSDB/opentsdb-tdigests">T-Digests</a>.</p> <p>Some additional features include:</p> <ul><li>HBase Date Tiered Compaction support to improve storage efficiency.<br/></li> <li>A new authentication plugin interface to support enterprise use cases.<br/></li> <li>An interface to support fetching data directly from Bigtable or HBase rows using a search index such as ElasticSearch. This improves queries for small subsets of high cardinality data and we’re working on open sourcing our code for the ES schema.<br/></li> <li>Greater UID cache controls and an optional LRU implementation to reduce the amount of JVM heap allocated to UID to string mappings.<br/></li> <li>Configurable query size and time limits to avoid OOMing a JVM with large queries.<br/></li> </ul><p>Try the releases on <a href="https://github.com/OpenTSDB/opentsdb/releases">GitHub</a> and let us know of any issues you run into by posting on GitHub issues or the <a href="https://groups.google.com/forum/#!forum/opentsdb">OpenTSDB Forum</a>. Your feedback is appreciated!</p> <p><b>OpenTSDB 3.0</b></p> <p>Additionally, we’ve started on 3.0, which is a rewrite that will support a slew of new features including:</p> <ul><li>Querying and analyzing data from the plethora of new time series stores.<br/></li> <li>A fully configurable query graph that allows for complex queries OpenTSDB 1x and 2x couldn’t support.<br/></li> <li>Streaming results to improve the user experience and avoid overwhelming a single query node.<br/></li> <li>Advanced analytics including support for time series forecasting with Yahoo’s <a href="https://github.com/yahoo/egads">EGADs</a> library.<br/></li> </ul><p>Please join us in testing out the current <a href="http://opentsdb.net/docs/3x/build/html/index.html">3.0</a> code, reporting bugs, and adding features.</p> </blockquote>