<p>This blog post describes Zedge’s use of Vespa for search and recommender systems to support content discovery for personalization of mobile phones (Android, iOS and Web). Zedge is now using Vespa in production to serve millions of monthly active users. See the architecture below.</p><figure data-orig-width="960" data-orig-height="720" class="tmblr-full"><img src="https://66.media.tumblr.com/4ff68800a3f000abee1d633c90420407/tumblr_inline_peh82w1D1i1vpfrlb_540.png" alt="image" data-orig-width="960" data-orig-height="720"/></figure><h2>What is Zedge?</h2> <figure data-orig-width="697" data-orig-height="584" class="tmblr-full"><img src="https://66.media.tumblr.com/ed34389c4d2f706b3151a47807f28dc1/tumblr_inline_peh84w5juQ1vpfrlb_540.png" alt="image" data-orig-width="697" data-orig-height="584"/></figure><p>Zedge’s main product is an app - <a href="https://play.google.com/store/apps/details?id=net.zedge.android&amp;hl=en">Zedge Ringtones &amp; Wallpapers</a> - that provides wallpapers, ringtones, game recommendations and notification sounds customized for your mobile device.  Zedge apps have been downloaded more than 300 million times combined for iOS and Android and is used by millions of people worldwide each month. Zedge is traded on NYSE under the ticker ZDGE.</p><p>People use Zedge apps for self-expression. Setting a wallpaper or ringtone on your mobile device is in many ways similar to selecting clothes, hairstyle or other fashion statements. In fact people try a wallpaper or ringtone in a similar manner as they would try clothes in a dressing room before making a purchase decision, they try different wallpapers or ringtones before deciding on one they want to keep for a while. </p><p>The decision for selecting a wallpaper is not taken lightly, since people interact and view their mobile device screen (and background wallpaper) a lot (hundreds of times per day).<br/></p> <h2>Why Zedge considered Vespa</h2> <p>Zedge apps - for iOS, Android and Web - depend heavily on search and recommender services to support content discovery. These services have been developed over several years and constituted of multiple subsystems - both internally developed and open source - and technologies for both search and recommender serving. In addition there were numerous big data processing jobs to build and maintain data for content discovery serving. The time and complexity of improving search and recommender services and corresponding processing jobs started to become high, so simplification was due. </p><figure data-orig-width="935" data-orig-height="77" class="tmblr-full"><img src="https://66.media.tumblr.com/311ff8d5cb16c502f55c6760a3fbe64d/tumblr_inline_peh86e5kcE1vpfrlb_540.png" alt="image" data-orig-width="935" data-orig-height="77"/></figure><p>Vespa seemed like a promising open source technology to consider for Zedge, in particular since it was proven in several ways within Oath (Yahoo):</p><ol><li><b>Scales to handle very large systems</b>, e.g. <br/></li><ol type="a"><li><a href="https://www.flickr.com/">Flickr</a> with billions of images and</li><li><a href="https://gemini.yahoo.com/advertiser/home">Yahoo Gemini Ads Platform</a> with more than one hundred thousand request per second to serve ads to 1 billion monthly active users for services such as Techcrunch, Aol, Yahoo!, Tumblr and Huffpost.</li></ol><li><b>Runs stable and requires very little operations support</b> - Oath has a few hundred - many of them large - Vespa based applications requiring less than a handful operations people to run smoothly. <br/></li><li><b>Rich set of features that Zedge could gain from using</b><br/></li><ol type="a"><li>Built-in tensor processing support could simplify calculation and serving of related wallpapers (images) &amp; ringtones/notifications (audio)</li><li>Built-in support of Tensorflow models to simplify development and deployment of machine learning based search and recommender ranking (at that time in development according to Oath). </li><li>Search Chains</li></ol><li><b>Help from core developers of Vespa</b><br/></li></ol><h2>The Vespa pilot project</h2> <p>Given the content discovery technology need and promising characteristics of Vespa we started out with a pilot project with a team of software engineers, SRE and data scientists with the goals of:</p><ol><li>Learn about Vespa from hands-on development <br/></li><li>Create a realistic proof of concept using Vespa in a Zedge app<br/></li><li>Get initial answers to key questions about Vespa, i.e. enough to decide to go for it fully<br/></li><ol type="a"><li>Which of today’s API services can it simplify and replace?</li><li>What are the (cloud) production costs with Vespa at Zedge’s scale? (OPEX)</li><li>How will maintenance and development look like with Vespa? (future CAPEX)</li><li>Which new (innovation) opportunities does Vespa give?</li></ol></ol><p>The result of the pilot project was successful - we developed a good proof of concept use of Vespa with one of our Android apps internally and decided to start a project transferring all recommender and search serving to Vespa. Our impression after the pilot was that the main benefit was by making it easier to maintain and develop search/recommender systems, in particular by reducing amount of code and complexity of processing jobs.<br/></p> <h2>Autosuggest for search with Vespa</h2> <p>Since autosuggest (for search) required both low latency and high throughput we decided that it was a good candidate to try for production with Vespa first. Configuration wise it was similar to regular search (from the pilot), but snippet generation (document summary) requiring access to document store was superfluous for autosuggest. </p> <p>A good approach for autosuggest was to:</p><ol><li>Make all document fields searchable with autosuggest of type (in-memory) attribute<br/></li><ol type="a"><li><a href="https://docs.vespa.ai/documentation/attributes.html">https://docs.vespa.ai/documentation/attributes.html </a></li><li><a href="https://docs.vespa.ai/documentation/reference/search-definitions-reference.html#attribute">https://docs.vespa.ai/documentation/reference/search-definitions-reference.html#attribute </a></li><li><a href="https://docs.vespa.ai/documentation/search-definitions.html">https://docs.vespa.ai/documentation/search-definitions.html</a> (basics)</li></ol><li>Avoid snippet generation and using the document store by overriding the document-summary setting in search definitions to only access attributes<br/></li><ol type="a"><li><a href="https://docs.vespa.ai/documentation/document-summaries.html">https://docs.vespa.ai/documentation/document-summaries.html</a> </li><li><a href="https://docs.vespa.ai/documentation/nativerank.html">https://docs.vespa.ai/documentation/nativerank.html</a></li></ol></ol><figure data-orig-width="336" data-orig-height="720" class="tmblr-full"><img src="https://66.media.tumblr.com/3c459a411a0b0c03530bffe548230776/tumblr_inline_peh87uqn7l1vpfrlb_540.png" alt="image" data-orig-width="336" data-orig-height="720"/></figure><p>The figure above illustrates the autosuggest architecture. When the user starts typing in the search field, we fire a query with the search prefix to the Cloudflare worker - which in case of a cache hit returns the result (possible queries) to the client. In case of a cache miss the Cloudflare worker forwards the query to our Vespa instance handling autosuggest.</p><p>Regarding external API for autosuggest we use <a href="https://blog.cloudflare.com/introducing-cloudflare-workers/">Cloudflare Workers</a> (supporting Javascript on V8 and later perhaps multiple languages with <a href="https://webassembly.org">Webassembly</a>) to handle API queries from Zedge apps in front of Vespa running in Google Cloud. This setup allow for simple close-to-user caching of autosuggest results. </p> <h2>Search, Recommenders and Related Content with Vespa</h2> <p>Without going into details we had several recommender and search services to adapt to Vespa. These services were adapted by writing custom Vespa searchers and in some cases search chains:</p><ul><li><a href="https://docs.vespa.ai/documentation/searcher-development.html">https://docs.vespa.ai/documentation/searcher-development.html</a> <br/></li><li><a href="https://docs.vespa.ai/documentation/chained-components.html">https://docs.vespa.ai/documentation/chained-components.html</a> <br/></li></ul><p>The main change compared to our old recommender and related content services was the degree of dynamicity and freshness of serving, i.e. with Vespa more ranking signals are calculated on the fly using Vespa’s tensor support instead of being precalculated and fed into services periodically. Another benefit of this was that the amount of computational (big data) resources and code for recommender &amp; related content processing was heavily reduced.</p> <h4>Continuous Integration and Testing with Vespa</h4> <p>A main focus was to make testing and deployment of Vespa services with continuous integration (see figure below). We found that a combination of Jenkins (or similar CI product or service) with Docker Compose worked nicely in order to test new Vespa applications, corresponding configurations and data (samples) before deploying to the staging cluster with Vespa on Google Cloud. This way we can have a realistic test setup - with Docker Compose - that is close to being exactly similar to the production environment (even at hostname level).</p><figure data-orig-width="960" data-orig-height="720" class="tmblr-full"><img src="https://66.media.tumblr.com/0bc7a423ff7bcc2b4bcdbcf265ee50be/tumblr_inline_peh89aLTGN1vpfrlb_540.png" alt="image" data-orig-width="960" data-orig-height="720"/></figure><h4>Monitoring of Vespa with Prometheus and Grafana</h4> <p>For monitoring we created a tool that continuously read Vespa metrics, stored them in Prometheus (a time series database) and visualized them them with Grafana. This tool can be found on <a href="https://github.com/vespa-engine/vespa_exporter">https://github.com/vespa-engine/vespa_exporter</a>. More information about Vespa metrics and monitoring:</p><ul><li><a href="https://docs.vespa.ai/documentation/reference/metrics-health-format.html">https://docs.vespa.ai/documentation/reference/metrics-health-format.html</a><br/></li><li><a href="https://docs.vespa.ai/documentation/jdisc/metrics.html">https://docs.vespa.ai/documentation/jdisc/metrics.html</a><br/></li><li><a href="https://docs.vespa.ai/documentation/operations/admin-monitoring.html">https://docs.vespa.ai/documentation/operations/admin-monitoring.html</a><br/></li></ul><figure data-orig-width="960" data-orig-height="720" class="tmblr-full"><img src="https://66.media.tumblr.com/a54f99cd826caf338430a192714df7dc/tumblr_inline_peh89qWRYe1vpfrlb_540.png" alt="image" data-orig-width="960" data-orig-height="720"/></figure><h2>Conclusion</h2> <p>The team quickly got up to speed with Vespa with its good documentation and examples, and it has been running like a clock since we started using it for real loads in production. But this was only our first step with Vespa - i.e. consolidating existing search and recommender technologies into a more homogeneous and easier to maintain form. </p><p>With Vespa as part of our architecture we see many possible paths for evolving our search and recommendation capabilities (e.g. machine learning based ranking such as integration with <a href="https://www.tensorflow.org/">Tensorflow</a> and <a href="https://onnx.ai/">ONNX</a>).</p> <p>Best regards,<br/>Zedge Content Discovery Team<br/></p>