Writing Lua scripts with meta
<p><a href="https://www.linkedin.com/in/sheridanrawlins/">Sheridan Rawlins</a>, Architect, <a href="https://www.yahooinc.com/">Yahoo</a></p>
<h1>Summary</h1>
<p>In any file ending in <code>.lua</code> with the executable bit set (<code>chmod a+x</code>), putting a “shebang” line like the following lets you run it, and even pass arguments to the script that won’t be swallowed by meta</p>
<p><code>hello-world.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
print("hello world")
</code></pre>
<p>Screwdriver’s <a href="https://github.com/screwdriver-cd/meta-cli">meta</a> tool is provided to every job, regardless of which image you choose.</p>
<p>This means that you can write Screwdriver <a href="https://docs.screwdriver.cd/user-guide/commands.html">commands</a> or helper scripts as <a href="https://www.lua.org/">Lua</a> programs.</p>
<p>It was inspired by (but unrelated to) <a href="https://github.com/etcd-io/bbolt">etcd’s bolt</a>, as meta is a key-value store of sorts, and its <a href="https://github.com/spacewander/boltcli">boltcli</a> which also provides a lua runner that interfaces with bolt.</p>
<h2>Example script or sd-cmd</h2>
<p><code>run.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
meta.set("a-plain-string-key", "somevalue")
meta.set("a-key-for-json-value", { name = "thename", num = 123, array = { "foo", "bar", "baz" } })
</code></pre>
<p><!-- more --></p>
<h1>What is included?</h1>
<ol><li>A <a href="https://www.lua.org/manual/5.1/">Lua 5.1</a> interpreter written in go (<a href="https://github.com/yuin/gopher-lua">gopher-lua</a>)</li>
<li><p>meta CLI commands are exposed as methods on the <code>meta</code> object</p>
<ul><li><p>meta get</p>
<pre><code class="lua">local foo_value = meta.get('foo')
</code></pre></li>
<li><p>meta set</p>
<pre><code class="lua">-- plain string
meta.set('key', 'value')`
-- json number
meta.set('key', 123)`
-- json array
meta.set('key', { 'foo', 'bar', 'baz' })`
-- json map
meta.set('key', { foo = 'bar', bar = 'baz' })`
</code></pre></li>
<li><p>meta dump</p>
<pre><code class="lua">local entire_meta_tree = meta.dump()
</code></pre></li>
</ul></li>
<li><p>Libraries (aka “modules”) included in <a href="https://github.com/vadv/gopher-lua-libs">gopher-lua-libs</a> - while there are many to choose from here, some highlights include:</p>
<ul><li><a href="https://argparse.readthedocs.io">argparse</a> - when writing scripts, this is a nice CLI parser inspired from the python one.</li>
<li>Encoding modules: <a href="https://github.com/vadv/gopher-lua-libs/tree/master/json">json</a>, <a href="https://github.com/vadv/gopher-lua-libs/tree/master/yaml">yaml</a>, and <a href="https://github.com/vadv/gopher-lua-libs/tree/master/base64">base64</a> allow you to decode or encode values as needed.</li>
<li>String helper modules: <a href="https://github.com/vadv/gopher-lua-libs/tree/master/strings">strings</a>, and <a href="https://github.com/vadv/gopher-lua-libs/tree/master/shellescape">shellescape</a></li>
<li><a href="https://github.com/vadv/gopher-lua-libs/tree/master/http">http client</a> - helpful if you want to use the <a href="https://api.screwdriver.cd/v4/documentation">Screwdriver REST API</a> possibly using <a href="https://www.lua.org/manual/5.1/manual.html#pdf-os.getenv">os.getenv</a> with the environment vars provided by screwdriver - <code>SD_API_URL</code>, <code>SD_TOKEN</code>, <code>SD_BUILD_ID</code> can be very useful.</li>
<li><a href="https://github.com/vadv/gopher-lua-libs/tree/master/plugin">plugin</a> - is an advanced technique for parallelism by firing up several “workers” or “threads” as “goroutines” under the hood and communicating via go channels. More than likely overkill for normal use-cases, but it may come in handy, such as fetching all artifacts from another job by reading its manifest.txt and fetching in parallel.</li>
</ul></li>
</ol><h1>Why is this interesting/useful?</h1>
<h2>meta is atomic</h2>
<p>When invoked, <code>meta</code> obtains an advisory lock via <a href="https://linux.die.net/man/1/flock">flock</a>.</p>
<p>However, if you wanted to <em>update</em> a value from the shell, you might perform two commands and lose the <code>atomicity</code>:</p>
<pre><code class="bash"># Note, to treat the value as an integer rather than string, use -j to indicate json
declare -i foo_count="$(meta get -j foo_count)"
meta set -j foo_count "$((++foo_count))"
</code></pre>
<p>While uncommon, if you write builds that do several things in parallel (perhaps a Makefile run with <code>make -j $(nproc)</code>), making such an update in parallel could hit race conditions between the get and set.</p>
<p>Instead, consider this script (or sd-cmd)</p>
<p><code>increment-key.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
local argparse = require 'argparse'
local parser = argparse(arg[0], 'increment the value of a key')
parser:argument('key', 'The key to increment')
local args = parser:parse()
local value = tonumber(meta.get(args.key)) or 0
value = value + 1
meta.set(args.key, value)
print(value)
</code></pre>
<p>Which can be run like so, and will be atomic</p>
<pre><code class="bash">./increment-key.lua foo
1
./increment-key.lua foo
2
./increment-key.lua foo
3
</code></pre>
<h2>meta is provided to every job</h2>
<p>The <code>meta</code> tool is made available to all builds, regardless of the image your build chooses - including minimal jobs intended for fanning in several jobs to a single one for further pipeline job-dependency graphs (i.e. <a href="https://hub.docker.com/r/screwdrivercd/noop-container">screwdrivercd/noop-container</a>)</p>
<p>Screwdrivers <a href="https://docs.screwdriver.cd/user-guide/commands.html">commands</a> can help share common tasks between jobs within an organization. When commands are written in <code>bash</code>, then any callouts it makes such as <a href="https://stedolan.github.io/jq/">jq</a> must either exist on the images or be installed by the sd-cmd. While writing in meta’s lua is not completely immune to needing “other things”, at least it has proper http and json support for making and interpreting REST calls.</p>
<h2>running “inside” meta can workaround system limits</h2>
<p>Occasionally, if the data you put into <code>meta</code> gets very large, you may encounter <a href="https://linux.die.net/man/2/execve">Limits on size of arguments and environment</a>, which comes from UNIX systems when invoking executables.</p>
<p>Imagine, for instance, wanting to put a file value into meta (<strong>NOTE</strong>: this is not a recommendation to put large things in meta, but, on the occasions where you need to, it can be supported). Say I have a file <code>foobar.txt</code> and want to put it into <code>some-key</code>. This code:</p>
<pre><code class="bash">foobar="$(< foobar.txt)"
meta set some-key "$foobar"
</code></pre>
<p>May fail to invoke <code>meta</code> at all if the args get too big.</p>
<p>If, instead, the contents are passed over redirection rather than an argument, this limit can be avoided:</p>
<p><code>load-file.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
local argparse = require 'argparse'
local parser = argparse(arg[0], 'load json from a file')
parser:argument('key', 'The key to put the json in')
parser:argument('filename', 'The filename')
local args = parser:parse()
local f, err = io.open(args.filename, 'r')
assert(not err, err)
local value = f:read("*a")
-- Meta set the key to the contents of the file
meta.set(args.key, value)
</code></pre>
<p>May be invoked with either the filename or, if the data is in memory with the named stdin device</p>
<pre><code class="bash"># Direct from the file
./load-file.lua some-key foobar.txt
# If in memory using "Here String" (<a href="https://www.gnu.org/software/bash/manual/bash.html#Here-Strings">https://www.gnu.org/software/bash/manual/bash.html#Here-Strings</a>)
foobar="$(< foobar.txt)"
./load-file.lua some-key /dev/stdin <<<"$foobar"
</code></pre>
<h1>Additional examples</h1>
<h2>Using http module to obtain the parent id</h2>
<p><code>get-parent-build-id.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
local http = require 'http'
local json = require 'json'
SD_BUILD_ID = os.getenv('SD_BUILD_ID') or error('SD_BUILD_ID environment variable is required')
SD_TOKEN = os.getenv('SD_TOKEN') or error('SD_TOKEN environment variable is required')
SD_API_URL = os.getenv('SD_API_URL') or error('SD_API_URL environment variable is required')
local client = http.client({ headers = { Authorization = "Bearer " .. SD_TOKEN } })
local url = string.format("%sbuilds/%d", SD_API_URL, SD_BUILD_ID)
print(string.format("fetching buildInfo from %s", url))
local response, err = client:do_request(http.request("GET", url))
assert(not err, err)
assert(response.code == 200, "error code not ok " .. response.code)
local buildInfo = json.decode(response.body)
print(tonumber(buildInfo.parentBuildId) or 0)
</code></pre>
<p>Invocation examples:</p>
<pre><code class="bash"># From a job that is triggered from another job
declare -i parent_build_id="$(./get-parent-build-id.lua)"
echo "$parent_build_id"
48242862
# From a job that is not triggered by another job
declare -i parent_build_id="$(./get-parent-build-id.lua)"
echo "$parent_build_id"
0
</code></pre>
<h2>Larger example to pull down manifests from triggering job in parallel</h2>
<p>This advanced script creates 3 argparse “commands” (manifest, copy, and parent-id) to help copying manifest files from parent job (the job that triggers this one).</p>
<p>it demonstrates advanced argparse features, http client, and the plugin module to create a “boss + workers” pattern for parallel fetches:</p>
<ul><li>Multiple workers fetch individual files requested by a work channel</li>
<li>The “boss” (main thread) filters relevent files from the manifest which it sends down the work channel</li>
<li>The “boss” closes the work channel, then waits for all workers to complete tasks (note that a channel will still deliver any elements before a <code>receive()</code> call reports <code>not ok</code></li>
</ul><p>This improves throughput considerably when fetching many files - from a worst case of the sum of all download times with one at a time, to a best case of just the maximum download time when all are done in parallel and network bandwidth is sufficient.</p>
<p><code>manifest.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
-- Imports
argparse = require 'argparse'
plugin = require 'plugin'
http = require 'http'
json = require 'json'
log = require 'log'
strings = require 'strings'
filepath = require 'filepath'
goos = require 'goos'
-- Parse the request
parser = argparse(arg[0], 'Artifact operations such as fetching manifest or artifacts from another build')
parser:option('-l --loglevel', 'Set the loglevel', 'info')
parser:option('-b --build-id', 'Build ID')
manifestCommand = parser:command('manifest', 'fetch the manifest')
manifestCommand:option('-k --key', 'The key to set information in')
copyCommand = parser:command('copy', 'Copy from and to')
copyCommand:option('-p --parallelism', 'Parallelism when copying multiple artifacts', 4)
copyCommand:flag('-d --dir')
copyCommand:argument('source', 'Source file')
copyCommand:argument('dest', 'Destination file')
parentIdCommand = parser:command("parent-id", "Print the parent-id of this build")
args = parser:parse()
-- Setup logs is shared with workers when parallelizing fetches
function setupLogs(args)
-- Setup logs
log.debug = log.new('STDERR')
log.debug:set_prefix("[DEBUG] ")
log.debug:set_flags { date = true }
log.info = log.new('STDERR')
log.info:set_prefix("[INFO] ")
log.info:set_flags { date = true }
-- TODO(scr): improve log library to deal with levels
if args.loglevel == 'info' then
log.debug:set_output('/dev/null')
elseif args.loglevel == 'warning' or args.loglevel == 'warning' then
log.debug:set_output('/dev/null')
log.info:set_output('/dev/null')
end
end
setupLogs(args)
-- Globals from env
function setupGlobals()
SD_API_URL = os.getenv('SD_API_URL')
assert(SD_API_URL, 'missing SD_API_URL')
SD_TOKEN = os.getenv('SD_TOKEN')
assert(SD_TOKEN, 'missing SD_TOKEN')
client = http.client({ headers = { Authorization = "Bearer " .. SD_TOKEN } })
end
setupGlobals()
-- Functions
-- getBuildInfo gets the build info json object from the buildId
function getBuildInfo(buildId)
if not buildInfo then
local url = string.format("%sbuilds/%d", SD_API_URL, buildId)
log.debug:printf("fetching buildInfo from %s", url)
local response, err = client:do_request(http.request("GET", url))
assert(not err, err)
assert(response.code == 200, "error code not ok " .. response.code)
buildInfo = json.decode(response.body)
end
return buildInfo
end
-- getParentBuildId gets the parent build ID from this build’s info
function getParentBuildId(buildId)
local parentBuildId = getBuildInfo(buildId).parentBuildId
assert(parentBuildId, string.format("could not get parendId for %d", buildId))
return parentBuildId
end
-- getArtifact gets and returns the requested artifact
function getArtifact(buildId, artifact)
local url = string.format("%sbuilds/%d/artifacts/%s", SD_API_URL, buildId, artifact)
log.debug:printf("fetching artifact from %s", url)
local response, err = client:do_request(http.request("GET", url))
assert(not err, err)
assert(response.code == 200, string.format("error code not ok %d for url %s", response.code, url))
return response.body
end
-- getManifestLines returns an iterator for the lines of the manifest and strips off leading ./
function getManifestLines(buildId)
return coroutine.wrap(function()
local manifest = getArtifact(buildId, 'manifest.txt')
local manifest_lines = strings.split(manifest, '\n')
for _, line in ipairs(manifest_lines) do
line = strings.trim_prefix(line, './')
if line ~= '' then
coroutine.yield(line)
end
end
end)
end
-- fetchArtifact fetches the artifact "source" and writes to a local file "dest"
function fetchArtifact(buildId, source, dest)
log.info:printf("Copying %s to %s", source, dest)
local sourceContent = getArtifact(buildId, source)
local dest_file = io.open(dest, 'w')
dest_file:write(sourceContent)
dest_file:close()
end
-- fetchArtifactDirectory fetches all the artifacts matching "source" from the manifest and writes to a folder "dest"
function fetchArtifactDirectory(buildId, source, dest)
-- Fire up workers to run fetches in parallel
local work_body = [[
http = require 'http'
json = require 'json'
log = require 'log'
strings = require 'strings'
filepath = require 'filepath'
goos = require 'goos'
local args, workCh
setupLogs, setupGlobals, fetchArtifact, getArtifact, args, workCh = unpack(arg)
setupLogs(args)
setupGlobals()
log.debug:printf("Starting work %p", _G)
local ok, work = workCh:receive()
while ok do
log.debug:print(table.concat(work, ' '))
fetchArtifact(unpack(work))
ok, work = workCh:receive()
end
log.debug:printf("No more work %p", _G)
]]
local workCh = channel.make(tonumber(args.parallelism))
local workers = {}
for i = 1, tonumber(args.parallelism) do
local worker_plugin = plugin.do_string(work_body,
setupLogs, setupGlobals, fetchArtifact, getArtifact, args, workCh)
local err = worker_plugin:run()
assert(not err, err)
table.insert(workers, worker_plugin)
end
-- Send workers work to do
log.info:printf("Copying directory %s to %s", source, dest)
local source_prefix = strings.trim_suffix(source, filepath.separator()) .. filepath.separator()
for line in getManifestLines(buildId) do
log.debug:print(line, source_prefix)
if source == '.' or source == '' or strings.has_prefix(line, source_prefix) then
local dest_dir = filepath.join(dest, filepath.dir(line))
goos.mkdir_all(dest_dir)
workCh:send { buildId, line, filepath.join(dest, line) }
end
end
-- Close the work channel to signal workers to exit
log.debug:print('Closing workCh')
err = workCh:close()
assert(not err, err)
-- Wait for workers to exit
log.debug:print('Waiting for workers to finish')
for _, worker in ipairs(workers) do
local err = worker:wait()
assert(not err, err)
end
log.info:printf("Done copying directory %s to %s", source, dest)
end
-- Normalize/help the buildId by getting the parent build id as a convenience
if not args.build_id then
SD_BUILD_ID = os.getenv('SD_BUILD_ID')
assert(SD_BUILD_ID, 'missing SD_BUILD_ID')
args.build_id = getParentBuildId(SD_BUILD_ID)
end
-- Handle the command
if args.manifest then
local value = {}
for line in getManifestLines(args.build_id) do
table.insert(value, line)
if not args.key then
print(line)
end
end
if args.key then
meta.set(args.key, value)
end
elseif args.copy then
if args.dir then
fetchArtifactDirectory(args.build_id, args.source, args.dest)
else
fetchArtifact(args.build_id, args.source, args.dest)
end
elseif args['parent-id'] then
print(getParentBuildId(args.build_id))
end
</code></pre>
<h3>Testing</h3>
<p>In order to test this, <a href="https://github.com/sstephenson/bats">bats testing system</a> was used to invoke <code>manifest.lua</code> with various arguments and the return code, output, and side-effects checked.</p>
<p>For unit tests, an http server was fired up to serve static files in a testdata directory, and manifest.lua was actually invoked within this <code>test.lua</code> file so that the http server and the manifest.lua were run in two separate threads (via the <code>plugin</code> module) but the same process (to avoid being blocked by meta’s locking mechanism, if run in two processes)</p>
<p><code>test.lua</code></p>
<pre><code class="lua">#!/usr/bin/env meta
-- Because Meta locks, run the webserver as a plugin in the same process, then invoke the actual file under test.
local plugin = require 'plugin'
local filepath = require 'filepath'
local argparse = require 'argparse'
local http = require 'http'
local parser = argparse(arg[0], 'Test runner that serves http test server')
parser:option('-d --dir', 'Dir to serve', filepath.join(filepath.dir(arg[0]), "testdata"))
parser:option('-a --addr', 'Address to serve on', "localhost:2113")
parser:argument('rest', "Rest of the args")
:args '*'
local args = parser:parse()
-- Run an http server on the requested (or default) addr and dir
local http_plugin = plugin.do_string([[
local http = require 'http'
local args = unpack(arg)
http.serve_static(args.dir, args.addr)
]], args)
http_plugin:run()
-- Wait for http server to be running and serve status.html
local wait_plugin = plugin.do_string([[
local http = require 'http'
local args = unpack(arg)
local client = http.client()
local url = string.format("http://%s/status.html", args.addr)
repeat
local response, err = client:do_request(http.request("GET", url))
until not err and response.code == 200
]], args)
wait_plugin:run()
-- Wait for it to finish up to 2 seconds
local err = wait_plugin:wait(2)
assert(not err, err)
-- With the http server running, run the actual file under test
-- Run with a plugin so that none of the plugins used by _this file_ are loaded before invoking dofile
local run_plugin = plugin.do_string([[
arg[0] = table.remove(arg, 1)
dofile(arg[0])
]], unpack(args.rest))
run_plugin:run()
-- Wait for the run to complete and report errors, if any
local err = run_plugin:wait()
assert(not err, err)
-- Stop the http server for good measure
http_plugin:stop()
</code></pre>
<p>And the bats test looked something like:</p>
<pre><code class="bash">#!/usr/bin/env bats
load test-helpers
function setup() {
mk_temp_meta_dir
export SD_META_DIR="$TEMP_SD_META_DIR"
export SD_API_URL="http://localhost:2113/"
export SD_TOKEN=SD_TOKEN
export SD_BUILD_ID=12345
export SERVER_PID="$!"
}
function teardown() {
rm_temp_meta_dir
}
@test "artifacts with no command is an error" {
run "${BATS_TEST_DIRNAME}/run.lua"
echo "$status"
echo "$output"
((status))
}
@test "manifest gets a few files" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" manifest
echo "$status"
echo "$output"
((!status))
grep foo.txt <<<"$output"
grep bar.txt <<<"$output"
grep manifest.txt <<<"$output"
}
@test "copy foo.txt myfoo.txt writes it properly" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" copy foo.txt "${TEMP_SD_META_DIR}/myfoo.txt"
echo "$status"
echo "$output"
((!status))
[[ $(<"${TEMP_SD_META_DIR}/myfoo.txt") == "foo" ]]
}
@test "copy bar.txt mybar.txt writes it properly" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" copy bar.txt "${TEMP_SD_META_DIR}/mybar.txt"
echo "$status"
echo "$output"
((!status))
[[ $(<"${TEMP_SD_META_DIR}/mybar.txt") == "bar" ]]
}
@test "copy -b 101010 -d somedir mydir writes it properly" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" -l debug copy -b 101010 -d somedir "${TEMP_SD_META_DIR}/mydir"
echo "$status"
echo "$output"
((!status))
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir"
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir" | grep one.txt
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir" | grep two.txt
(( $(<"${TEMP_SD_META_DIR}/mydir/somedir/one.txt") == 1 ))
(( $(<"${TEMP_SD_META_DIR}/mydir/somedir/two.txt") == 2 ))
}
@test "copy -b 101010 -d . mydir gets all artifacts" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" -l debug copy -b 101010 -d . "${TEMP_SD_META_DIR}/mydir"
echo "$status"
echo "$output"
((!status))
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir"
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir" | grep one.txt
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir" | grep two.txt
(( $(<"${TEMP_SD_META_DIR}/mydir/somedir/one.txt") == 1 ))
(( $(<"${TEMP_SD_META_DIR}/mydir/somedir/two.txt") == 2 ))
[[ $(<"${TEMP_SD_META_DIR}/mydir/abc.txt") == abc ]]
[[ $(<"${TEMP_SD_META_DIR}/mydir/def.txt") == def ]]
(($(find "${TEMP_SD_META_DIR}/mydir" -type f | wc -l) == 5))
}
@test "copy -b 101010 -d . -p 1 mydir gets all artifacts" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" -l debug copy -b 101010 -d . -p 1 "${TEMP_SD_META_DIR}/mydir"
echo "$status"
echo "$output"
((!status))
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir"
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir" | grep one.txt
ls -1 "${TEMP_SD_META_DIR}/mydir/somedir" | grep two.txt
(( $(<"${TEMP_SD_META_DIR}/mydir/somedir/one.txt") == 1 ))
(( $(<"${TEMP_SD_META_DIR}/mydir/somedir/two.txt") == 2 ))
[[ $(<"${TEMP_SD_META_DIR}/mydir/abc.txt") == abc ]]
[[ $(<"${TEMP_SD_META_DIR}/mydir/def.txt") == def ]]
(($(find "${TEMP_SD_META_DIR}/mydir" -type f | wc -l) == 5))
}
@test "parent-id 12345 gets 99999" {
run "${BATS_TEST_DIRNAME}/test.lua" -- "${BATS_TEST_DIRNAME}/run.lua" parent-id -b 12345
echo "$status"
echo "$output"
((!status))
(( $output == 99999 ))
}
</code></pre>