<p><a href="https://www.linkedin.com/in/sheridanrawlins/">Sheridan Rawlins</a>, Architect, <a href="https://www.verizonmedia.com">Verizon Media</a></p> <h2>Improving Screwdriver’s meta tool</h2> <p>Over the past month there have been a few changes to the meta tool mostly focused on using external metadata, but also on helping to identify and self-diagnose a few silent gotchas we found.</p> <p><a href="https://docs.screwdriver.cd/user-guide/metadata">Metadata</a> is a structured key/value data store that gives you access to details about your <a href="https://docs.screwdriver.cd/about/appendix/domain.html#build">build</a>. Metadata is carried over to all builds part of the same event, At the end of each build metadata is merged back to the event the build belongs to. This allows builds to share their metadata with other builds in the same event or external.</p> <h2>External metadata</h2> <p>External metadata can be populated for a job in your pipeline using the <code>requires</code> clause that refers to it in the form <code>sd@${pipelineID}:${jobName}</code> (such as <code>sd@12345:main</code>).</p> <p>If sd@12345:main runs to completion and “triggers” a job or jobs in your pipeline, a file will be created with meta from that build in <code>/sd/meta/sd@12345:main.json</code>, and you can refer to any of its data with commands such as <code>meta get someKey --external sd@12345:main</code>.</p> <p>The above feature has existed for some time, but there were several corner cases that made it challenging to use external metadata:</p> <ol><li>External metadata was not provided when the build was not triggered from the external pipeline such as by clicking the “Start” button, or via a scheduled trigger (i.e., through using the <a href="https://docs.screwdriver.cd/user-guide/configuration/annotations#job-level-annotations">buildPeriodically annotation</a>).</li> <li>Restarting a previously externally-triggered build would not provide the external metadata. The notion of <a href="https://docs.screwdriver.cd/user-guide/FAQ#how-do-i-rollback">rollback</a> can be facilitated by retriggering a deployment job, but if that deployment relies on metadata from an external trigger, it wouldn’t be there before these improvements.</li> </ol><h2>Fetching from lastSuccessfulMeta</h2> <p>Screwdriver has an API endpoint called <a href="https://api.screwdriver.cd/v4/documentation#!/v4/getV4JobsIdLastsuccessfulmeta">lastSuccessfulMeta</a>. While it is possible to use this endpoint directly, providing this ability directly in the <code>meta</code> tool makes it a little easier to just “make it so”. By default now, if external metadata does not exist in the file <code>/sd/meta/sd@12345:main.json</code>, it is fetched from that job’s lastSuccessfulMeta via the API call. Should this behavior not be desired, the flag <code>--skip-fetch</code> can be used to skip fetching.</p> <p>For rollback behavior, however, this feature by itself wasn’t enough - consider a good deployment followed by a bad deployment. The “bad” deployment would most likely have deployed what, from a build standpoint, was “successful”. When retriggering the previous job, because it is a manual trigger, there will be no external metadata and the lastSuccessfulMeta will most likely be fetched and the newer, “bad” code would just get re-deployed again.</p> <p>For this reason the next feature was also added to meta - “caching external data in the local meta”.</p> <h2>Caching external data in the local meta</h2> <p>External metadata for <code>sd@12345:main</code> (whether from trigger or fetched from lastSuccessfulMeta) will now be stored into and searched first from the local meta under the key <code>sd.12345.main</code>. Note: no caching will be done when <code>--skip-fetch</code> is passed.</p> <p>This caching of external meta helps with a few use cases:</p> <ol><li>Rollback is facilitated because the external metadata at the time a job was first run is now saved and used when “Restart” is pressed for a job.</li> <li>External metadata is now available throughout all jobs of an <a href="https://docs.screwdriver.cd/about/appendix/domain.html#event">event</a> - previously, only the triggered job or jobs would receive the external metadata, but because the local meta is propagated to every job in an <a href="https://docs.screwdriver.cd/about/appendix/domain.html#event">event</a>, the <code>sd.12345.main</code> key will be available to all jobs. Since meta will look there first, any job in a workflow can use that same <code>--external sd@12345:main</code> with confidence that it will get the same metadata which was received by the triggered job.</li> </ol><h2>Self-diagnosing gotchas</h2> <ol><li>Meta uses a CLI parser called <a href="https://github.com/urfave/cli/blob/master/README.md">urfave/cli</a>. Previously, it configured its CLI flags in both the “global” and “subcommand-specific” locations; this led to being able to pass flags in either location: before the subcommand like get or set or after it. However, they would only be honored in the latter position. Instead, only the &ndash;meta-space is global, and all other flags are per-subcommand. It is no longer possible to pass <code>--external</code> to the set subcommand.</li> <li>Number of arguments - previously, if extra arguments were passed to flags that didn’t take them, or if arguments were forgotten to flags that expected an argument, then it was possible to become confused about the key and/or value vs flags. Now, the flags are strictly counted - 1 (“key”) for get, and 2 - (“key”, “value”) for set.</li> </ol>