Introducing Queue Service
<p>Pritam Paul, Software Engineer, Verizon Media</p><p>We have recently made changes to the underlying Screwdriver Architecture for build processing. Previously, the <a href="https://github.com/screwdriver-cd/executor-queue"></a><a href="https://github.com/screwdriver-cd/executor-queue">executor-queue</a> was tightly-coupled to the SD API and worked by constantly polling for messages at specific intervals. Due to this design, the queue would block API requests. Furthermore, if the API crashed, scheduled jobs might not be added to the queue, causing cascading failures. </p><p>Hence, keeping the principles of isolation-of-concerns and abstraction in mind, we designed a more resilient REST-API-based queueing system: the Queue Service. This new service reads, writes and deletes messages from the queue after processing. It also encompasses the former capability of the <a href="https://github.com/screwdriver-cd/queue-worker">queue-worker</a> and acts as a scheduler.</p><h2>Authentication</h2><p>The SD API and Queue Service communicate bidirectionally using signed JWT tokens sent via auth headers of each request.</p><h2>Build Sequence</h2><figure data-orig-width="3038" data-orig-height="1129" class="tmblr-full"><img src="https://66.media.tumblr.com/d77f8f1c5b3c22d0c49a87c95477395e/625eb863e88bf54d-55/s540x810/0265a1a11f1d9bf6129bfa1904ec8e2d8cbd7f53.png" alt="image" data-orig-width="3038" data-orig-height="1129"/></figure><h2>Design Document</h2><p>For more details, check out our <a href="https://github.com/screwdriver-cd/screwdriver/blob/master/design/queue-service.md">design spec</a>.</p><h2>Using Queue Service</h2><p>As a cluster admin, to configure using the queue as an executor, you can deploy the queue-service as a REST API using a <a href="https://github.com/screwdriver-cd/queue-service/blob/master/screwdriver.yaml">screwdriver.yaml</a> and update configuration in SD API to point to the new service endpoint:</p><blockquote><p># config/default.yaml<br/>ecosystem:<br/> # Externally routable URL for the User Interface<br/> ui: <a href="https://cd.screwdriver.cd">https://cd.screwdriver.cd</a></p><p> # Externally routable URL for the Artifact Store<br/> store: <a href="https://store.screwdriver.cd">https://store.screwdriver.cd</a></p><p> # Badge service (needs to add a status and color)<br/> badges: <a href="https://img.shields.io/badge/build--.svg">https://img.shields.io/badge/build–.svg</a></p><p> # Internally routable FQDNS of the queue service<br/> queue: <a href="http://sdqueuesvc.screwdriver.svc.cluster.local">http://sdqueuesvc.screwdriver.svc.cluster.local</a></p><p>executor:<br/> plugin: queue<br/> queue: “</p></blockquote><p>For more configuration options, see the <a href="https://docs.screwdriver.cd/cluster-management/configure-queue-service">queue-service documentation</a>.</p><h2>Compatibility List</h2><p>In order to use the new workflow features, you will need these minimum versions:</p><ul><li>UI - v1.0.502</li><li>API - v0.5.887</li><li>Launcher - v6.0.56</li><li>Queue-Service - v1.0.11</li></ul><h2>Contributors</h2><p>Thanks to the following contributors for making this feature possible:</p><ul><li><a href="https://github.com/adong">adong</a></li><li><a href="https://github.com/klu909">klu909</a></li><li><a href="https://github.com/jithine">jithine</a></li><li><a href="https://github.com/parthasl">parthasl</a></li><li><a href="https://github.com/pritamstyz4ever">pritamstyz4ever</a></li><li><a href="https://github.com/tkyi">tkyi</a></li></ul><p><b>Questions and Suggestions</b></p><p>We’d love to hear from you. If you have any questions, please feel free to reach out<a href="https://docs.screwdriver.cd/about/support"> here</a>. You can also visit us on<a href="https://github.com/screwdriver-cd"> Github</a> and<a href="https://slack.screwdriver.cd/"> Slack</a>.</p>