# Introduction This document describes the acceptance criteria for the Radicle CI broker, as well as how to verify that they are met. Acceptance criteria here means a requirement that must be met for the software to be acceptable to its stakeholders. This file is used by [Subplot](https://subplot.tech/) to generate and run test code as part of running `cargo test`. # Stakeholders For the purposes of this document, a stakeholder is someone whose opinion matters for setting acceptance criteria. The CI broker has the following stakeholders, grouped so that specific people only need to be named in one place: * `cib-devs` – the people who develop the CI broker itself - Lars Wirzenius * `adapter-devs` – the people who develop adapters - Lars Wirzenius - Michalis - Yorgos Saslis * `node-ops` – the people operating a Radicle node, when they also run Radicle CI on it - Lars Wirzenius - Yorgos Saslis * `devs` – the people for whose repositories Radicle CI runs; this means the people who contribute to any repository hosted on Radicle, when any node runs CI for that repository, as opposed to the people who develop the Radicle CI software - Lars Wirzenius - Michalis Some stakeholders are named explicitly so that it will be easier to ask them more information that is captured in this document. Note that the list will evolve over time. Please suggest missing stakeholders to the developers and maintainers of the CI broker. # Data files shared between scenarios ## Broker configuration ~~~{#broker.yaml .file .yaml} db: ci-broker.db report_dir: reports default_adapter: mcadapterface queue_len_interval: 1min adapters: mcadapterface: command: ./adapter.sh config: foo: bar config_env: RADICLE_NATIVE_CI env: PATH: /bin sensitive_env: API_KEY: xyzzy filters: - !Branch "main" ~~~ ~~~{#broker-with-triggers.yaml .file .yaml} db: ci-broker.db report_dir: reports queue_len_interval: 1min adapters: mcadapterface: command: ./adapter.sh env: RADICLE_NATIVE_CI: native-ci.yaml sensitive_env: API_KEY: xyzzy triggers: - adapter: mcadapterface filters: - !Branch "main" ~~~ ~~~{#broker-with-all-filter-kinds.yaml .file .yaml} db: ci-broker.db report_dir: reports queue_len_interval: 1min adapters: mcadapterface: command: ./adapter.sh env: RADICLE_NATIVE_CI: native-ci.yaml sensitive_env: API_KEY: xyzzy triggers: - adapter: mcadapterface filters: - !And - !NoneOf - !Branch "main" - !TagCreated - !Or - Allow - !AnyOf - BranchCreated - !Not - BranchDeleted - !BranchUpdated - !TagCreated - !TagDeleted - !TagUpdated - !Branch "a2dec1a5b3ab5b34cea16c07b632023d9ce535fc" - !DefaultBranch - !Deny - !HasFile "xyzzy" - !Node "z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV" - !PatchCreated - !PatchUpdated - !Patch "a2dec1a5b3ab5b34cea16c07b632023d9ce535fc" - !Repository "rad:zwTxygwuz5LDGBq255RA2CbNGrz8" ~~~ ~~~{#broker-with-two-triggers.yaml .file .yaml} db: ci-broker.db report_dir: reports queue_len_interval: 1min adapters: mcadapterface: command: ./adapter.sh env: RADICLE_NATIVE_CI: native-ci.yaml sensitive_env: API_KEY: xyzzy triggers: - adapter: mcadapterface filters: - !Branch "main" - adapter: mcadapterface filters: - !Branch "main" ~~~ ## A dummy adapter This adapter does nothing, just reports a run ID and a successful run. Note that this adapter always outputs a message to its standard error output, even though it doesn't fail. This is useful for verifying that the CI broker logs adapter error output, and doesn't harm other uses of the adapter. ~~~{#dummy.sh .file .sh} #!/bin/sh set -eu cat > /dev/null echo '{"response":"triggered","run_id":{"id":"xyzzy"}}' # Simulate a CI run that takes a while. sleep 2 echo '{"response":"finished","result":"success"}' ( echo "This is an adapter error: Mordor" echo "Environment:" env if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then echo "Adapter config:" nl "$RADICLE_NATIVE_CI" fi ) 1>&2 ~~~ ## A failing adapter with a successful run This adapter does nothing, just reports a run ID and a successful run, but then fails. ~~~{#failing-on-success.sh .file .sh} #!/bin/sh set -eu cat > /dev/null echo '{"response":"triggered","run_id":{"id":"xyzzy"}}' echo '{"response":"finished","result":"success"}' ( echo "This is an adapter error: Mordor" echo "Environment:" env if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then echo "Adapter config:" nl "$RADICLE_NATIVE_CI" fi ) 1>&2 exit 1 ~~~ ## A failing adapter with a failed run This adapter does nothing, just reports a run ID and a failed run, but then fails. ~~~{#failing-on-failure.sh .file .sh} #!/bin/sh set -eu cat > /dev/null echo '{"response":"triggered","run_id":{"id":"xyzzy"}}' echo '{"response":"finished","result":"failure"}' ( echo "This is an adapter error: Mordor" echo "Environment:" env if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then echo "Adapter config:" nl "$RADICLE_NATIVE_CI" fi ) 1>&2 exit 1 ~~~ ## List job COBs Job COBs are a way for the CI broker to record that it's run CI for a change. This script lists the job COBs in a given repository. ~~~{#list-jobs.sh .file .sh} #!/bin/bash set -euo pipefail RID="$(rad ls --all | awk -v R="$1" '$2 == R { print $3 }')" if [ -z "$RID" ]; then echo "Unknown repository $1" 1>&2 exit 1 fi rad cob list --repo "$RID" --type xyz.radworks.job ~~~ # Custom scenario steps In this document we use scenarios to show how to verify that the CI broker does what we expect of it. For this, we define several custom scenario steps. In this chapter we describe those steps, and also verify that the steps work. ## Set up a node This step creates a Radicle node, the Radicle CI broker, and a CI adapter. > `given a Radicle node, with CI configured with {config} and adapter {adapter}` The captured parts of the step are: * `config` — the name of the embedded file (somewhere in this document) with the configuration for the CI broker * `adapter` — the name of the embedded file with the CI adapter implementation; we use simple shell script dummy adapter implementations, as in this document we only care about the broker/adapter interface, not that the adapter actually performs a CI run This step installs binaries (or makes them available to be run), and creates some files. It doesn't not start long-lived processes, in particular not the Radicle node process. We verify that this scenario works by examining the results. For clarity, we split the scenario into many snippets. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh ~~~ The programs we'll need are available to run. To check this, we use a helper shell script to verify that. This avoids us to work around limitations in Subplot for command parsing: Subplot does not parse steps the way the shell does, so there is no way to pass text that contains space characters to command as a single argument. ~~~scenario given file which.sh when I run bash which.sh rad when I run bash which.sh cib when I run bash which.sh cibtool when I run bash which.sh synthetic-events then command is successful ~~~ ~~~{#which.sh .file .sh} #!/bin/sh # We use Bash build-in command as that's portable. "which" is not. command -v "$1" ~~~ The configuration file must now exist. ~~~scenario then file broker.yaml exists ~~~ The adapter is to be installed as `adapter.sh` and it must be executable. ~~~scenario then file adapter.sh exists when I run ls -l adapter.sh then stdout matches regex ^-rwx ~~~ There is a Radicle home directory. ~~~scenario then directory .radicle exists then directory .radicle/keys exists then file .radicle/keys/radicle exists then file .radicle/keys/radicle.pub exists then directory .radicle/storage exists then file .radicle/config.json exists ~~~ We also need way to set up environment variables for commands we run, especially for `rad` to use the right node. Subplot does not have built in support for this (at least not yet), but we work around that by creating a shell script `env.sh` that sets them up. ~~~scenario then file env.sh exists when I run ls -l env.sh then stdout matches regex ^-rwx when I run ./env.sh env then stdout matches regex ^PATH= then stdout matches regex ^HOME= then stdout matches regex ^RAD_HOME= then stdout matches regex ^RAD_PASSPHRASE= then stdout matches regex ^RAD_SOCKET= ~~~ ## Create a repository This step creates a Git repository and makes it into a Radicle repository. > `given a Git repository {name} in the Radicle node` The captured part of the step is: * `name` — the Git and Radicle repository name We run the step and look at the results. We need the node creation step first. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository reppy in the Radicle node ~~~ The Git repository must exist. ~~~scenario then directory reppy exists then directory reppy/.git exists then file reppy/file.dat exists then file reppy/foobar does not exist when I run, in reppy, git show then stdout matches regex ^commit ~~~ It must also be a Radicle repository and in the local node. ~~~scenario when I run ./env.sh rad ls --all then stdout contains "reppy" ~~~ ## Queue a node event for processing This step queues a node event to be processed later by the `synthetic-events` test helper tool that is part of the CI broker. The step does this by creating a fake `refsUpdated` node event and writing that to file with a specific name. > `given the Radicle node emits a refsUpdated event for {repodir}` The captured part of the step is: * `repodir` — the directory where the repository is for which the event is created To set up this step, we need to have node and a repository first. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository reppy in the Radicle node given the Radicle node emits a refsUpdated event for reppy ~~~ We check that the event file looks roughly correct by querying it with the `jq` tool. ~~~scenario then file event.json exists when I run jq .type event.json then stdout contains ""refsFetched"" ~~~ This is a very rudimentary check, but if the event file is incorrect, then Radicle code will reject it. We don't want to duplicate the logic to do that verification in detail. # Acceptance criteria ## Shows YAML config as JSON _Want:_ The CI broker can write out the configuration it uses at run time as JSON. _Why:_ This is helpful for the node operator to verify that they have configured the program correctly. _Who:_ `cib-devs` Our verification here is quite simplistic, and only checks that the output is in the JSON format. It does not try to make sure the JSON matches the YAML semantically. ~~~scenario given a Radicle node, with CI configured with broker-with-all-filter-kinds.yaml and adapter dummy.sh when I run cib --config broker-with-all-filter-kinds.yaml config --output actual.json when I run jq . actual.json then command is successful ~~~ ## Shows JSON config as JSON _Want:_ The CI broker can write out the configuration it usesc at run time as JSON. _Why:_ This is helpful for the node operator to verify that they have configured the program correctly. _Who:_ `cib-devs` Our verification here is quite simplistic, and only checks that the output is in the JSON format. It does not try to make sure the JSON matches the YAML semantically. ~~~scenario given a Radicle node, with CI configured with config.json and adapter dummy.sh when I run cib --config config.json config --output actual.json when I run jq . actual.json then command is successful ~~~ ~~~{#config.json .file .json} { "default_adapter": "mcadapterface", "triggers": null, "report_dir": "reports", "db": "ci-broker.db", "adapters": { "mcadapterface": { "command": "./adapter.sh", "env": { "PATH": "/bin" }, "sensitive_env": { "API_KEY": "" }, "config": { "foo": "bar" }, "config_env": "RADICLE_NATIVE_CI" } }, "filters": [ { "Branch": "main" } ], "max_run_time": "1min", "queue_len_interval": "1s", "concurrent_adapters": null } ~~~ ## Shows adapter specification _Want:_ The CI broker can write out the specification for an adapter. _Why:_ This is helpful for the node operator to verify that they have specified the adapter correctly. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I run cib --config broker.yaml adapters --output adapters.json when I run jq . adapters.json then command is successful then stdout contains ""foo": "bar"" then stdout contains ""config_env": "RADICLE_NATIVE_CI"" ~~~ ## Refuses config with an unknown field _Want:_ The CI broker refused to load a configuration file that has unknown fields. _Why:_ This is helpful for detecting typos and other mistakes in configuration files instead of ignoring them silently. _Who:_ `cib-devs`, `node-ops` ~~~scenario given a Radicle node, with CI configured with buggy.yaml and adapter dummy.sh when I try to run cib --config buggy.yaml config then command fails then stderr contains "xyzzy" then stderr contains "unknown field" ~~~ ~~~{#buggy.yaml .file .yaml} db: ci-broker.db report_dir: reports queue_len_interval: 1min adapters: mcadapterface: command: ./adapter.sh env: RADICLE_NATIVE_CI: native-ci.yaml sensitive_env: API_KEY: xyzzy triggers: - adapter: mcadapterface filters: - !Branch "main" xyzzy: "this field is unknown" ~~~ ## Smoke test: Runs adapter _Want:_ CI broker can run its adapter. _Why:_ This is obviously necessary. If this doesn't work, nothing else has a hope of working. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt given a directory reports given file list-jobs.sh when I run bash list-jobs.sh xyzzy then stdout is exactly "" when I run ./env.sh cib --config broker-with-triggers.yaml process-events then stderr contains "CibStart" then stderr contains "CibConfig" then stderr contains "CibEndSuccess" then file reports/index.html exists then file reports/status.json exists then file reports/index.rss exists when I run bash list-jobs.sh xyzzy then stdout isn't exactly "" when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list --json then stdout contains ""id": "xyzzy"" ~~~ ## Processes empty event queue successfully _Want:_ CI broker does nothing, but successfully, if asked to process an empty event queue. _Why:_This is an important corner case for `cib queued`. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter dummy.sh when I run ./env.sh cib --config broker-with-triggers.yaml queued when I run cibtool --db ci-broker.db run list then stdout is exactly "" ~~~ ## Handles adapter failing on a successful run _Want:_ If the adapter fails, the CI broker creates a job COB and report pages anyway. _Why:_ This is necessary for the CI broker to be robust. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter failing-on-success.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt given a directory reports given file list-jobs.sh when I run bash list-jobs.sh xyzzy then stdout is exactly "" when I run ./env.sh cib --config broker-with-triggers.yaml process-events then stderr contains "CibStart" then stderr contains "CibConfig" then stderr contains "CibEndSuccess" then file reports/index.html exists then file reports/status.json exists then file reports/index.rss exists when I run bash list-jobs.sh xyzzy then stdout isn't exactly "" when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list --json then stdout contains ""id": "xyzzy"" ~~~ ## Handles adapter failing on a failed run _Want:_ If the adapter fails, the CI broker creates a job COB and report pages anyway. _Why:_ This is necessary for the CI broker to be robust. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter failing-on-failure.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt given a directory reports given file list-jobs.sh when I run bash list-jobs.sh xyzzy then stdout is exactly "" when I run ./env.sh cib --config broker-with-triggers.yaml process-events then stderr contains "CibStart" then stderr contains "CibConfig" then stderr contains "CibEndSuccess" then file reports/index.html exists then file reports/status.json exists then file reports/index.rss exists when I run bash list-jobs.sh xyzzy then stdout isn't exactly "" when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list --json then stdout contains ""id": "xyzzy"" ~~~ ## Runs adapter with configuration _Want:_ CI broker can run its adapter and give it the configuration in the CI broker adapter specification. _Why:_ Being able to embed the adapter configuration in the `cib` configuration file makes is more convenient for the node operators to specify different adapter configurations for different purposes. _Who:_ `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt given a directory reports when I run ./env.sh cib --config broker.yaml process-events then stderr contains "RADICLE_NATIVE_CI=" then stderr contains "foo: bar" when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list --json then stdout contains ""id": "xyzzy"" ~~~ ## Runs adapter without a report directory _Want:_ CI broker can run without a report directory. _Why:_ We don't require the report directory to be specified, or exist, but we do require `cib` to handle this. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run ./env.sh cib --config broker-with-triggers.yaml process-events then stderr contains "CibStart" then stderr contains "CibConfig" then stderr contains "CibEndSuccess" when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list --json then stdout contains ""id": "xyzzy"" ~~~ ## Runs adapters for all matching triggers _Want:_ CI broker can run its adapter. _Why:_ This is obviously necessary. If this doesn't work, nothing else has a hope of working. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker-with-two-triggers.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt given a directory reports when I run ./env.sh cib --config broker-with-two-triggers.yaml process-events then stderr contains "CibStart" then stderr contains "CibConfig" then stderr contains "CibEndSuccess" then file reports/index.html exists then file reports/status.json exists then file reports/index.rss exists when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list then stdout has 2 lines containing "xyzzy" ~~~ ## Runs adapter on each type of event _Want:_ CI broker runs the adapter for each type of CI event. _Why:_ The adapter needs to handle each type of CI event. _Who:_ `cib-devs` We verify this by adding CI events to the event queue using `cibtool` and checking that `cib` can process those. This is simpler and more direct than emitting node events that result in the desired CI events. We are here not concerned about whether `cib` handles node events or turns those into the correct CI events: we verify that in other ways. We first set things up, including creating a repository `xyzzy`, and a Radicle patch in that repository. The id of the patch is in the file `patch-id.txt` so that it can be used. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a directory reports when I run ./env.sh env -C xyzzy git switch -c branchy given file create-patch when I run ./env.sh env -C xyzzy bash ../create-patch ../patch-id.txt ~~~ Verify that `cib` can process a branch creation event. ~~~ when I run rm -f ci-broker.db when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-created --id-file id.txt when I run ./env.sh cib --config broker.yaml queued when I run cibtool --db ci-broker.db run list then stdout has one line ~~~ Verify that `cib` can process a branch update event. ~~~ when I run rm -f ci-broker.db when I run cibtool --db ci-broker.db event add --repo xyzzy --ref brancy --base main --kind branch-updated --id-file id.txt when I run ./env.sh cib --config broker.yaml queued when I run cibtool --db ci-broker.db run list then stdout has one line ~~~ Verify that `cib` can process a branch deletion event. ~~~ when I run rm -f ci-broker.db when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted --id-file id.txt when I run ./env.sh cib --config broker.yaml queued when I run cibtool --db ci-broker.db run list then stdout has one line ~~~ Verify that `cib` can process a patch creation event. ~~~ when I run rm -f ci-broker.db when I run cibtool --db ci-broker.db event add --repo xyzzy --kind patch-created --patch-id-file patch-id.txt --id-file id.txt when I run ./env.sh cib --config broker.yaml queued when I run cibtool --db ci-broker.db run list then stdout has one line ~~~ Verify that `cib` can process a patch update event. ~~~ when I run rm -f ci-broker.db when I run cibtool --db ci-broker.db event add --repo xyzzy --kind patch-updated --patch-id-file patch-id.txt --id-file id.txt when I run ./env.sh cib --config broker.yaml queued when I run cibtool --db ci-broker.db run list then stdout has one line ~~~ ~~~{#create-patch .file .sh} #!/bin/sh set -eu touch foo git add foo git commit -m foo EDITOR=/bin/true git push rad HEAD:refs/patches rad patch list | awk 'NR == 4 { print $3 }' | xargs rad patch show | awk 'NR == 3 { print $3 }' >"$1" ~~~ ## Reports it version _Want:_ `cib` and `cibtool` report their version, if invoked with the `--version` potion. _Why:_ This helps node operators include the version in any bug reports. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I run cib --version then stdout matches regex ^radicle-ci-broker \d+\.\d+\.\d+@ when I run cibtool --version then stdout matches regex ^radicle-ci-broker \d+\.\d+\.\d+@ ~~~ ## Adapter can provide URL for info on run _Want:_ The adapter can provide a URL for information about the run, such a run log. This optional. _Why:_ The CI broker does not itself store the run log, but it's useful to be able to point users at one. The CI broker can put that into a Radicle COB or otherwise store it so that users can see it. Note, however, that the adapter gets to decide which URL to provide: it need not be the run log. It might, for example, be a URL to the web view of a "pipeline" in GitLab CI instead, from which the user can access individual logs. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter adapter-with-url.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I try to run ./env.sh cib --config broker.yaml insert then command is successful ~~~ ~~~{#adapter-with-url.sh .file .sh} #!/bin/sh set -eu echo '{"response":"triggered","run_id":{"id":"xyzzy"},"info_url":"https://ci.example.com/xyzzy"}' echo '{"response":"finished","result":"success"}' ~~~ ## Gives helpful error message if node socket can't be found _Want:_ If the CI broker can't connect to the Radicle node control socket, it gives an error message that helps the user to understand the problem. _Why:_ This helps users deal with problems themselves and reduces the support burden on the Radicle project. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I try to run ./env.sh cib --config broker.yaml insert then command fails then stderr contains "node control socket does not exist: synt.sock" ~~~ ## Gives helpful error message if it doesn't understand its configuration file _Want:_ If the CI broker is given a configuration file that it can't understand, it gives an error message that explains the problem to the user. _Why:_ This helps users deal with problems themselves and reduces the support burden on the Radicle project. _Who:_ `cib-devs` _Comment:_ This is a very basic scenario. Error handling is by nature a thing that can always be made better. We can later add more scenarios if we tighten the acceptance criteria. ~~~scenario given a Radicle node, with CI configured with not-yaml.yaml and adapter dummy.sh when I try to run cib --config not-yaml.yaml config then command fails then stderr contains "failed to parse configuration file as YAML: not-yaml.yaml" ~~~ ~~~{#not-yaml.yaml .file} This file is not YAML. ~~~ ## Stops if the node connection breaks _Want:_ If the connection to the Radicle node, via its control socket, breaks, the CI broker terminates with a message saying why. _Why:_ The CI broker can either keep running and trying to re-connect, or it can terminate. Either is workable. However, it's a simpler design and less code to terminate and allow re-starting to be handled by a dedicated system, such as `systemd`. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I try to run ./env.sh cib --config broker.yaml insert then command is successful ~~~ ## Can shut down cleanly _Want:_ The node operator can instrut the running CI broker to shut down after currently executing runs finish. _Why:_This is useful for both the CI broker test suite and for operators to shut down service for maintenance. _Who:_ `cib-devs`, `node-ops` We verify this by starting a relatively long-running CI run and also telling `cib` to shut down, and then verifying the CI run finished successfully. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run cibtool --db ci-broker.db event shutdown when I try to run ./env.sh cib --config broker.yaml queued then command is successful when I run cibtool --db ci-broker.db run list --json then stdout contains "success" ~~~ ## Produces a report page upon request _Want:_ The node operator can run a command to produce a report of all CI runs a CI broker instance has performed. _Why:_ This is useful for diagnosis, if nothing else. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a directory reports when I run ./env.sh cibtool --db x.db run add --repo xyzzy --branch main --commit HEAD --failure when I run ./env.sh cibtool --db x.db report --output-dir reports then file reports/index.html exists then file reports/index.html contains "xyzzy" ~~~ This doesn't check that there is a per-repository HTML file, because we have not convenient way to know the repository ID. ## Logs adapter stderr output _What:_ The CI broker should log, to its own log output, the adapter's stderr output. _Why:_ This allows the adapter to output its own log to its standard error output. This makes it easier to debug adapter problems. _Who:_ `adapter-devs`, `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter broken-adapter.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run ./env.sh cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run ./env.sh cibtool --db ci-broker.db event list --json given a directory reports when I run ./env.sh cib --config broker.yaml queued then stderr contains "Rivendell" then stderr contains "Mordor" ~~~ This adapter outputs a broken response message, and after that something to its stderr. The CI broker is meant to read and log both. ~~~{#broken-adapter.sh .file .sh} #!/bin/sh set -eu cat > /dev/null echo '{"response":"Rivendell"}}' echo "This is an adapter error: Mordor" 1>&2 ~~~ ## Allows setting minimum log level _What:_ The node admin should be able to set the minimum log level for log messages that get output to stderr. _Why:_ This allows controlling how much log spew log admins have to see. _Who:_ `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I run ./env.sh cib --config broker.yaml config then stderr contains "CibStart" when I run ./env.sh cib --config broker.yaml --log-level error config then stderr is exactly "" ~~~ ## Fails run if building trigger fails, but does not crash _Want:_ The CI broker fails a CI run if it can't create a trigger message from a CI event, but it continues running and processing other events. _Why:_ If it's not possible to create a trigger message, the CI run can't succeed, unless the failure is temporary. However, we have no way of knowing if the failure is temporary, so the safe thing is to mark the CI run as having failed and removing the CI event from the queue. Further, the CI broker should not crash and should process other events. _Who:_ `cib-dev` A failure to create a trigger message happens if the CI event refers to a repository, commit, or Git ref that doesn't exist in the repository on the local node. This should not ever happen, as the CI event is only emitted by the node after the changes are on the node. However, it has happened due to a programming error in the CI broker. By handling the error and removing the event, the CI broker is a little bit more robust. We verify this by inserting two events into the queue and then running `cib queued` to process them. We arrange things so that the first event fails, but the second one succeeds. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a directory reports when I run ./env.sh cibtool --db ci-broker.db event add --repo xyzzy --commit f42ea5cb9ce2dc7a9b87834ccee5b9bb3867db90 --kind branch-updated --base main when I run ./env.sh cibtool --db ci-broker.db event add --repo xyzzy --kind branch-updated --base main when I run ./env.sh cib --config broker.yaml queued then stderr contains "f42ea5cb9ce2dc7a9b87834ccee5b9bb3867db90" when I run cibtool --db ci-broker.db run list then stdout has one line ~~~ # Acceptance criteria for event filtering The scenarios in this chapter verify that the event filters work as intended. Each scenario sets up the event queue with an event, and runs `cib queued` to process the event queue, and then verifies that CI was run, or not run, as appropriate. In each scenario we verify by running CI twice: once to make sure the filter allows what it should, and once to make sure it doesn't allow what it shouldn't ## Filter predicate `Repository` _Want:_ We can allow an event that is for a specific repository. _Why:_ We want to constrain CI to a specific repository. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a Git repository other in the Radicle node given file config.yaml from filter-repository.yaml given file update-repoid.sh when I run bash update-repoid.sh xyzzy config.yaml when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run cibtool --db ci-broker.db trigger --repo other --commit HEAD when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains ""repo_name": "xyzzy"" then stdout doesn't contain ""repo_name": "other"" ~~~ ~~~{#filter-repository.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Repository "REPOID" ~~~ ~~~{#update-repoid.sh .file .sh} #!/bin/sh set -eu dir="$1" yaml="$2" rid="$(cd "$dir" && rad .)" sed -i "s/REPOID/$rid/g" "$yaml" ~~~ ## Filter predicate `Node` _Want:_ We can allow an event that originates in a given node. _Why:_ We want to constrain CI to a specific developer. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a Git repository other in the Radicle node given file config.yaml from filter-node.yaml given file update-nodeid.sh when I run bash update-nodeid.sh xyzzy config.yaml when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run cibtool --db ci-broker.db trigger --repo other --commit HEAD --node z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains ""repo_name": "xyzzy"" then stdout doesn't contain ""repo_name": "other"" ~~~ ~~~{#filter-node.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Node "NODEID" ~~~ ~~~{#update-nodeid.sh .file .sh} #!/bin/sh set -eu dir="$1" yaml="$2" rid="$(cd "$dir" && rad self --nid)" sed -i "s/NODEID/$rid/g" "$yaml" ~~~ ## Filter predicate `AnyDelegate` _Want:_ We can allow an event that originates in a node for any delegate. _Why:_ We want to constrain CI to privileged developers for a repository. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a Git repository other in the Radicle node given file config.yaml from filter-anydelegate.yaml given file update-nodeid.sh when I run bash update-nodeid.sh xyzzy config.yaml when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run cibtool --db ci-broker.db trigger --repo other --commit HEAD --node z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains ""repo_name": "xyzzy"" then stdout doesn't contain ""repo_name": "other"" ~~~ ~~~{#filter-anydelegate.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !AnyDelegate ~~~ ## Filter predicate `Tag` _Want:_ We can allow an event that is about a specific tag. _Why:_ We want to constrain CI to specific tags, such as for releases. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-tag.yaml when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout doesn't contain ""repo_name": "xyzzy"" when I try to run cibtool --db ci-broker.db trigger --repo xyzzy --commit v1.0 then command fails when I run env -C xyzzy git tag -am "version 1.0" v1.0 when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout doesn't contain ""repo_name": "xyzzy"" ~~~ ~~~{#filter-tag.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Tag "v\\d+(\\.\\d+)" ~~~ ## Filter predicate `Branch` _Want:_ We can allow an event that is about a specific branch. _Why:_ We want to constrain CI to specific branches, such as the `main` branch. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given a Git repository other in the Radicle node given file config.yaml from filter-branch.yaml when I run cibtool --db ci-broker.db trigger --repo xyzzy --commit HEAD when I run cibtool --db ci-broker.db trigger --repo other --commit HEAD --ref oksa when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains ""repo_name": "xyzzy"" then stdout doesn't contain ""repo_name": "other"" ~~~ ~~~{#filter-branch.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Branch "main" ~~~ ## Filter predicate `BranchCreated` _Want:_ We can allow an event for a branch having been created. _Why:_ We want to constrain CI to only new branches. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-branchcreated.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-created when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-updated --base main when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains ""main"" then stdout doesn't contain "oksa" ~~~ ~~~{#filter-branchcreated.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !BranchCreated ~~~ ## Filter predicate `BranchUpdated` _Want:_ We can allow an event for a branch having been updated. _Why:_ We want to constrain CI to only updated branches, as distinct from new branches. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-branchupdated.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-updated --base main when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" then stdout doesn't contain "oksa" ~~~ ~~~{#filter-branchupdated.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !BranchUpdated ~~~ ## Filter predicate `BranchDeleted` _Want:_ We can allow an event for a branch having been deleted. _Why:_ We want to constrain CI to only deleted branches, e.g., to update a mirror. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-branchdeleted.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" then stdout doesn't contain "oksa" ~~~ ~~~{#filter-branchdeleted.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !BranchDeleted ~~~ ## Filter predicate `Allow` _Want:_ We can allow all events. _Why:_ This is for consistency. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-allow.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" then stdout contains "oksa" ~~~ ~~~{#filter-allow.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Allow ~~~ ## Filter predicate `Deny` _Want:_ We can allow no events. _Why:_ This is for consistency. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-deny.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout doesn't contain "main" then stdout doesn't contain "oksa" ~~~ ~~~{#filter-deny.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Deny ~~~ ## Filter predicate `And` _Want:_ We can allow a combination of events if they are all allowed individually. _Why:_ This is for consistency. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-and.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" then stdout contains "oksa" ~~~ ~~~{#filter-and.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !And - !Allow - !Allow ~~~ ## Filter predicate `Or` _Want:_ We can allow a combination of events if any of them are allowed individually. _Why:_ This is for consistency. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-or.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" then stdout contains "oksa" ~~~ ~~~{#filter-or.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Or - !Allow - !Deny ~~~ ## Filter predicate `Not` _Want:_ We can allow an event if the contained filter denies it. _Why:_ This is for consistency. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-not.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-deleted when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout doesn't contain "main" then stdout doesn't contain "oksa" ~~~ ~~~{#filter-not.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !Not - !Allow ~~~ ## Filter predicate `DefaultBranch` _Want:_ We can allow an event if the event refers to the default branch. _Why:_ This is so that the user doesn't need to spell out the name explicitly. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-defaultbranch.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-created when I run cibtool --db ci-broker.db event add --repo xyzzy --ref oksa --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" then stdout doesn't contain "oksa" ~~~ ~~~{#filter-defaultbranch.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !DefaultBranch ~~~ ## Filter predicate `HasFile` _Want:_ We can allow an event if its commit contains a file or directory by this name. _Why:_ This is so that the user can choose a suitable adapter. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given file config.yaml from filter-hasfile-missing.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout doesn't contain "main" given file config.yaml from filter-hasfile.yaml when I run cibtool --db ci-broker.db event add --repo xyzzy --kind branch-created when I run ./env.sh cib --config config.yaml queued when I run cibtool --db ci-broker.db run list --json then stdout contains "main" ~~~ ~~~{#filter-hasfile.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !HasFile "file.dat" ~~~ ~~~{#filter-hasfile-missing.yaml .file .json} db: ci-broker.db adapters: default: command: ./adapter.sh triggers: - adapter: default filters: - !HasFile "does-not-exist" ~~~ # Acceptance criteria for test tooling The event synthesizer is a helper to feed the CI broker node events in a controlled fashion. ## We can run rad _Want:_ We can run rad. _Why:_ For many of the verification scenarios for the CI broker we need to run the Radicle `rad` command line tool. Depending on the environment we use for verification, `rad` may be installed in various places. Commonly, if installed using the Radicle installer, `rad` is installed into `~/.radicle/bin` and edits the shell initialization file to add that to `$PATH`. However, in a CI context, that initialization is not necessarily done and so the `radenv.sh` helper script adds that directory to the `$PATH` just in case. We verify in this scenario that we can run `rad` at all. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I try to run rad --version then command is successful ~~~ ## Dummy adapter runs successfully _Want:_ The dummy adapter (in embedded file `dummy.sh`) runs successfully. _Why:_ Test scenarios using the dummy adapter need to be able to rely that it works. _Who:_ `cib-devs` ~~~scenario given file dummy.sh when I run chmod +x dummy.sh when I try to run ./dummy.sh then command is successful ~~~ ## Adapter with URL runs successfully _Want:_ The adapter with a URL (in embedded file `adapter-with-url.sh`) runs successfully. _Why:_ Test scenarios using this adapter need to be able to rely that it works. _Who:_ `cib-devs` ~~~scenario given file adapter-with-url.sh when I run chmod +x adapter-with-url.sh when I try to run ./adapter-with-url.sh then command is successful ~~~ ## Event synthesizer terminates after first connection _Want:_ The event synthesizer runs in the background, but terminates after the first connection. _Why:_ This is needed so that it can be invoked in Subplot scenarios. _Who:_ `cib-devs` We use the `synthetic-events --client` option to connect to the daemon and wait for the daemon to delete the socket file. This is more easily portable than using a generic tool such as `nc`, which has many variants across operating systems. We wait for up to ten seconds the `synthetic-events` daemon to remove the socket file before we check that it's been deleted, but checking for that every second. This avoids the trap of waiting for a fixed time: if the time is too short, the scenario fails spuriously, and if it's very long, the scenario takes longer than necessary. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh then file synt.sock does not exist when I run synthetic-events --log daemon.log synt.sock then file synt.sock exists when I run synthetic-events --client --log daemon.log synt.sock then file synt.sock does not exist ~~~ # Acceptance criteria for persistent database The CI broker uses an SQLite database for persistent data. Many processes may need to access or modify the database at the same time. While SQLite is good at managing that, it needs to be used in the right way for everything to work correctly. The acceptance criteria in this chapter address that. To enable the verification of these acceptance criteria, the CI broker database allows for a "counter", as a single row in a dedicated table. Concurrency is tested by having multiple processes update the counter at the same time and verifying the end result is as intended and that every value is set exactly once. ## Count in a single process _Want:_ A single process can increment the test counter correctly. _Why:_ If this doesn't work with a single process, it won't work of multiple processes, either. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh then file count.db does not exist when I run cibtool --db count.db counter show then stdout is exactly "0\n" when I run cibtool --db count.db counter count --goal 1000 when I run cibtool --db count.db counter show then stdout is exactly "1000\n" ~~~ ## Insert events into queue _Want:_ Insert broker events generated from node events into persistent event queue in the database, when allowed by the CI broker event filter. _Why:_ This is fundamental for running CI when repositories in a node change. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I try to run ./env.sh env cib --config broker.yaml insert then command is successful when I run cibtool --db ci-broker.db event list --json then stdout contains "BranchUpdated" then stdout contains ""branch": "main"" then stdout contains ""tip":" then stdout contains ""old_tip":" ~~~ ## Insert many events into queue _Want:_ Insert many events that arrive quickly. _Why:_ We need at least some rudimentary performance testing. _Who:_ `cib-devs` when I run synthetic-events synt.sock refsfetched.json --log synt.log --repeat 1000 ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt --repeat 1000 when I try to run ./env.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful when I run cibtool --db ci-broker.db event count then stdout is exactly "1000\n" ~~~ ## Process queued events _Want:_ It's possible to run the CI broker in a mode where it only processes events from its persistent event queue. _Why:_ This is primarily useful for testing the CI broker queuing implementation. _Who:_ `cib-devs` We verify this by adding events to the queue with `cibtool`, and then running the CI broker and verifying it terminates after processing the events. We carefully add a shutdown event so that the CI broker shuts down. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository testy in the Radicle node when I run ./env.sh cibtool --db ci-broker.db event add --repo testy --kind branch-updated --base main when I run cibtool --db ci-broker.db event shutdown given a directory reports when I run ./env.sh cib --config broker.yaml queued then stderr contains "QueueProcActionRun" then stderr contains "QueueProcActionShutdown" when I run cibtool --db ci-broker.db event list then stdout is empty when I run cibtool --db ci-broker.db run list --json then stdout contains "success" ~~~ ## Count in concurrent processes _Want:_ Two process can concurrently increment the test counter correctly. _Why:_ This is necessary, if not necessarily sufficient, for concurrent database use to work correctly. _Who:_ `cib-devs` Due to limitations in Subplot we mange the concurrent processes using a helper shell script,k `count.sh`, found below. It runs two concurrent `cibtool` processes that update the same database file, and count to a desired goal. The script then verifies that everything went correctly. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given file count.sh when I run bash -x count.sh 100 10 then stdout contains "OK\n" ~~~ ~~~{#count.sh .file .sh} #!/bin/sh set -eu run() { cibtool --db "$DB" counter count --goal "$goal" } DB=count.db goal="$1" reps="$2" for x in $(seq "$reps"); do echo "Repetition $x" rm -f "$DB" ./?.out run >1.out 2>&1 & one=$! run >2.out 2>&1 & two=$! if ! wait "$one"; then echo "first run failed" cat 1.out exit 1 fi if ! wait "$two"; then echo "second run failed" cat 2.out exit 1 fi if grep ERROR ./?.out; then echo found ERRORs exit 1 fi n="$(sqlite3 "$DB" 'select counter from counter_test')" [ "$n" == "$goal" ] || ( echo "wrong count $n" exit 1 ) if awk '/increment to/ { print $NF }' ./?.out | sort -n | uniq -d | grep .; then echo "duplicate increments" exit 1 fi done echo OK ~~~ # Acceptance criteria for management tool The `cibtool` management tool can be used to examine and change the CI broker database, and thus indirectly manage what the CI broker does. ## Events can be queued and removed from queue _Want:_ `cibtool` can show the queued events, can inject an event, and remove an event. _Why:_ This is the minimum functionality needed to manage the event queue. _Who:_ `cib-devs` We verify that this works by adding a new broker event, and then removing it. We randomly choose the repository id for the CI broker itself for this test, but the id shouldn't matter, it just needs to be of the correct form. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run cibtool --db x.db event list then stdout is empty when I run ./env.sh cibtool --db x.db event add --repo xyzzy --base HEAD --id-file id.txt --kind branch-updated when I run cibtool --db x.db event show --id-file id.txt then stdout contains "rad:" then stdout contains "main" when I run cibtool --db x.db event remove --id-file id.txt when I run cibtool --db x.db event list then stdout is empty ~~~ ## Can remove all queued events _Want:_ `cibtool` can remove all queued events in one operation. _Why:_ This will be useful if the CI broker changes how CI events or their serialization in an incompatible way, again, or when the node operator wants to prevent many CI runs from happening. _Who:_ `cib-devs`, `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository testy in the Radicle node when I run cibtool --db x.db event list then stdout is empty when I run ./env.sh cibtool --db x.db event add --repo testy --base HEAD --kind branch-updated when I run ./env.sh cibtool --db x.db event add --repo testy --base HEAD --kind branch-updated when I run ./env.sh cibtool --db x.db event add --repo testy --base HEAD --kind branch-updated when I run ./env.sh cibtool --db x.db event add --repo testy --base HEAD --kind branch-updated when I run ./env.sh cibtool --db x.db event add --repo testy --base HEAD --kind branch-updated when I run ./env.sh cibtool --db x.db event add --repo testy --base HEAD --kind branch-updated when I run cibtool --db x.db event remove --all when I run cibtool --db x.db event list then stdout is empty ~~~ ## Can add shutdown event to queue _Want:_ `cibtool` can add a shutdown event to the queued events. _Why:_ This is needed for testing, and for the node operator to be able to do this cleanly. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I run cibtool --db x.db event list then stdout is empty when I run cibtool --db x.db event shutdown --id-file id.txt when I run cibtool --db x.db event show --id-file id.txt then stdout contains "Shutdown" ~~~ ## Can add a branch creation event to queue _Want:_ `cibtool` can add an event for branch being created to the queued events. _Why:_ This is needed for testing. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh env -C xyzzy git switch -c oksa when I run ./env.sh cibtool --db x.db event add --repo xyzzy --ref oksa --kind branch-created when I run cibtool --db x.db event list --json then stdout contains "BranchCreated" ~~~ ## Can add a branch update event to queue _Want:_ `cibtool` can add an event for branch being updated to the queued events. _Why:_ This is needed for testing. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh env -C xyzzy git switch -c oksa when I run ./env.sh cibtool --db x.db event add --repo xyzzy --ref oksa --kind branch-updated --base HEAD when I run cibtool --db x.db event list --json then stdout contains "BranchUpdated" ~~~ ## Can add a branch deletion event to queue _Want:_ `cibtool` can add an event for branch being deleted to the queued events. _Why:_ This is needed for testing. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh env -C xyzzy git switch -c oksa when I run ./env.sh cibtool --db x.db event add --repo xyzzy --ref oksa --kind branch-deleted when I run cibtool --db x.db event list --json then stdout contains "BranchDeleted" ~~~ ## Can add a patch creation event to queue _Want:_ `cibtool` can add an event for a branch being created to the queued events. _Why:_ This is needed for testing. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh env -C xyzzy git switch -c oksa when I run ./env.sh cibtool --db x.db event add --repo xyzzy --ref oksa --kind patch-created --patch-id f863364f6774160607d90811b06a0e401c097466 when I run cibtool --db x.db event list --json then stdout contains "PatchCreated" ~~~ ## Can add a patch update event to queue _Want:_ `cibtool` can add an event for a branch being updated to the queued events. _Why:_ This is needed for testing. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh env -C xyzzy git switch -c oksa when I run ./env.sh cibtool --db x.db event add --repo xyzzy --ref oksa --kind patch-updated --patch-id f863364f6774160607d90811b06a0e401c097466 when I run cibtool --db x.db event list --json then stdout contains "PatchUpdated" ~~~ ## Can trigger a CI run _Want:_ The node operator can easily trigger a CI run without changing the repository. _Why:_ This allows running CI on a schedule, for example. It's also useful for CI broker development. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh cibtool --db x.db trigger --repo xyzzy --commit HEAD --id-file id.txt when I run cibtool --db x.db event show --id-file id.txt then stdout contains "rad:" then stdout contains "main" ~~~ ## Can output trigger message for a CI run _Want:_ The `cibtool` command can output the CI event to trigger a CI run to the standard output or a file. _Why:_ This is helpful for debugging the CI broker at least. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run ./env.sh cibtool --db x.db trigger --repo xyzzy --commit HEAD --stdout then stdout contains "rad:" then stdout contains "main" when I run ./env.sh cibtool --db x.db trigger --repo xyzzy --commit HEAD --output trigger.json then file trigger.json contains "rad:" then file trigger.json contains "main" when I run cibtool --db x.db event list then stdout is empty ~~~ ## Add information about triggered run to database _Want:_ `cibtool` can add information about a triggered CI run. _Why:_ This is primarily needed for testing. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run cibtool --db x.db run list then stdout is empty when I run ./env.sh cibtool --db x.db run add --id runny --repo xyzzy --branch main --commit main --triggered when I run cibtool --db x.db run list --json then stdout contains "runny" then stdout contains ""state": "triggered"" ~~~ ## Add information about run that's running to database _Want:_ `cibtool` can add information about a CI run that's running. _Why:_ This is primarily needed for testing. _Who:_ `cib-dev`. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run cibtool --db x.db run list then stdout is empty when I run ./env.sh cibtool --db x.db run add --repo xyzzy --url https://x/1 --branch main --commit HEAD --running when I run cibtool --db x.db run list --json then stdout contains ""repo_name": "xyzzy"" then stdout contains ""state": "running"" ~~~ ## Add information about run that's finished successfully to database _Want:_ `cibtool` can add information about a CI run that's finished successfully. _Why:_ This is primarily needed for testing. _Who:_ `cib-dev`. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node when I run cibtool --db x.db run list then stdout is empty when I run ./env.sh cibtool --db x.db run add --id runny --repo xyzzy --branch main --commit HEAD --success when I run cibtool --db x.db run list --json then stdout contains ""state": "finished"" then stdout contains ""result": "success"" then stdout contains "xyzzy" ~~~ ## Add information about run that's finished in failure to database _Want:_ `cibtool` can add information about a CI run that's failed. _Why:_ This is primarily needed for testing. _Who:_ `cib-dev`. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository testy in the Radicle node when I run cibtool --db x.db run list then stdout is empty when I run ./env.sh cibtool --db x.db run add --id xyzzy --repo testy --branch main --commit HEAD --failure when I run cibtool --db x.db run list --json then stdout contains ""state": "finished"" then stdout contains ""result": "failure"" then stdout contains "xyzzy" when I run cibtool --db x.db run list --adapter-run-id abracadabra then stdout is empty ~~~ ## Remove information about a run from the database _Want:_ `cibtool` can removed information about a CI run. _Why:_ This is primarily for completeness. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run cibtool --db x.db run list then stdout is empty when I run ./env.sh cibtool --db x.db run add --id runny --repo xyzzy --branch main --commit main --triggered when I run cibtool --db x.db run list --json then stdout contains "runny" then stdout contains ""state": "triggered"" when I run cibtool --db x.db run remove --adapter runny when I run cibtool --db x.db run list then stdout is empty ~~~ ## Update and show information about run to running _Want:_ `cibtool` can update information about a CI run. _Why:_ This is primarily needed for testing. _Who:_ `cib-dev`. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository testy in the Radicle node when I run cibtool --db x.db run list then stdout is empty when I run ./env.sh cibtool --db x.db run add --id x --repo testy --branch main --commit HEAD --triggered when I run cibtool --db x.db run list then stdout has one line when I run cibtool --db x.db run show x then stdout contains ""state": "triggered"" then stdout contains ""result": null" when I run cibtool --db x.db run update --id x --running when I run cibtool --db x.db run list then stdout has one line when I run cibtool --db x.db run show x then stdout contains ""state": "running"" then stdout contains ""result": null" when I run cibtool --db x.db run update --id x --success when I run cibtool --db x.db run list then stdout has one line when I run cibtool --db x.db run show x then stdout contains ""state": "finished"" then stdout contains ""result": "success"" when I run cibtool --db x.db run update --id x --failure when I run cibtool --db x.db run list then stdout has one line when I run cibtool --db x.db run show x then stdout contains ""state": "finished"" then stdout contains ""result": "failure"" ~~~ ## Don't insert event for non-existent repository _Want:_ `cibtool` won't insert an event to the queue for a repository that isn't in the local node. _Why:_ This prevents adding events that can't ever trigger a CI run. _Who:_ `cib-devs` Note that we verify both lookup by name and by repository ID, and by `cibtool event add` and `cibtool trigger`, to cover all the cases. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository testy in the Radicle node when I try to run ./env.sh cibtool --db x.db event add --repo missing --base c0ffee --kind branch-updated then command fails then stderr contains "missing" when I try to run ./env.sh cibtool --db x.db event add --repo rad:z3byzFpcfbMJBp4tKYyuuTZiP8WUB --base c0ffee --kind branch-updated then command fails then stderr contains "rad:z3byzFpcfbMJBp4tKYyuuTZiP8WUB" when I try to run ./env.sh cibtool --db x.db trigger --repo missing --commit HEAD --id-file id.txt then command fails then stderr contains "missing" when I try to run ./env.sh cibtool --db x.db trigger --repo rad:z3byzFpcfbMJBp4tKYyuuTZiP8WUB --commit HEAD --id-file id.txt then command fails then stderr contains "rad:z3byzFpcfbMJBp4tKYyuuTZiP8WUB" ~~~ ## Record node events _What:_ Node operator can record node events into a file. _Why:_ This can be helpful for remote debugging, it's very helpful for CI broker development to see what events actually happen, and it's useful for gathering data for trying out event filters. _Who:_ `cib-devs`, `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run ./env.sh cibtool event record --output events.json then file events.json contains ""type":"refsFetched"" ~~~ ## Convert recorded node events into CI events _What:_ Node operator can see what CI events are created from node events. _Why:_ This is helpful so that node operators can see what CI events are created from node events, which may have been previously recorded. It's also helpful for CI broker developers as a development tool. _Who:_ `cib-dev`, `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run ./env.sh cibtool event record --output node-events.json when I run ./env.sh cibtool event ci --output ci-events.json node-events.json when I run cat ci-events.json then file ci-events.json contains "BranchUpdated"" ~~~ ## Filter recorded CI events _What:_ Node operator can see what CI events an event filter allows. _Why:_ This is helpful so that node operators can see verify their event filters work as they expect. _Who:_ `cib-dev`, `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository xyzzy in the Radicle node given the Radicle node emits a refsUpdated event for xyzzy when I run ./env.sh synthetic-events synt.sock event.json --log log.txt when I run ./env.sh cibtool event record --output node-events.json when I run ./env.sh cibtool event ci --output ci-events.json node-events.json given file allow.yaml when I run cibtool event filter allow.yaml ci-events.json then stdout contains "BranchUpdated" given file deny.yaml when I run cibtool event filter deny.yaml ci-events.json then stdout is empty ~~~ ~~~{#allow.yaml .file .yaml} filters: - !Branch "main" ~~~ ~~~{#deny.yaml .file .yaml} filters: - !Branch "this-does-not-exist" ~~~ ## Extract `cib` log from journald and pretty print _Want:_ `cibtool` can extract `cib` log messages from the systemd journal sub-system, and pretty print them, and optionally filter the messages. _Why:_ systemd is the common service manager for Linux systems, and it needs to be convenient to extract `cib` log messages from its system logging sub-system, journald. This is especially important for CI broker developers who need to diagnose problems on remote `cib` instances to which they don't have direct access. _Who:_ `cib-devs` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given file journal.json when I run cibtool log --format journald journal.json --output x.json when I run jq . x.json then command is successful when I run cibtool log --format journald journal.json --broker-run-id 62c45727-a4d8-4a29-9dae-88c6e8b61655 --output x.json when I run grep -c timestamp x.json then stdout has one line ~~~ ~~~{#journal.json .file .json} {"__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31360;b=da4ba18425ea4e34bd0f0731f0270572;m=c3256a61;t=6290db5b772ca;x=280fbc307405cb81","_SYSTEMD_UNIT":"radicle-ci-broker.service","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_SYSTEMD_SLICE":"system.slice","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_PID":"526","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:00.276073Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_RUNTIME_SCOPE":"system","_CAP_EFFECTIVE":"0","_GID":"1001","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","__MONOTONIC_TIMESTAMP":"3274009185","PRIORITY":"6","_COMM":"cib","_UID":"1001","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","_HOSTNAME":"radicle-ci","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","__REALTIME_TIMESTAMP":"1733988720276170","SYSLOG_IDENTIFIER":"cib","_TRANSPORT":"stdout","_EXE":"/usr/bin/cib","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","SYSLOG_FACILITY":"3"} {"_GID":"1001","_RUNTIME_SCOPE":"system","_UID":"1001","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_PID":"526","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_CAP_EFFECTIVE":"0","_SELINUX_CONTEXT":"unconfined\n","_TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"cib","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:01.276463Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_SYSTEMD_SLICE":"system.slice","__MONOTONIC_TIMESTAMP":"3275009597","PRIORITY":"6","SYSLOG_FACILITY":"3","_HOSTNAME":"radicle-ci","_EXE":"/usr/bin/cib","_COMM":"cib","_SYSTEMD_UNIT":"radicle-ci-broker.service","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31361;b=da4ba18425ea4e34bd0f0731f0270572;m=c334ae3d;t=6290db5c6b6a5;x=a9c49aef9ad2a509","__REALTIME_TIMESTAMP":"1733988721276581"} {"__MONOTONIC_TIMESTAMP":"3276010022","SYSLOG_IDENTIFIER":"cib","_CAP_EFFECTIVE":"0","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_HOSTNAME":"radicle-ci","_EXE":"/usr/bin/cib","_UID":"1001","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_SYSTEMD_SLICE":"system.slice","_PID":"526","SYSLOG_FACILITY":"3","_COMM":"cib","PRIORITY":"6","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_SELINUX_CONTEXT":"unconfined\n","_TRANSPORT":"stdout","_SYSTEMD_UNIT":"radicle-ci-broker.service","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:02.276881Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31362;b=da4ba18425ea4e34bd0f0731f0270572;m=c343f226;t=6290db5d5fa8e;x=843418c04ac1932f","_GID":"1001","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","__REALTIME_TIMESTAMP":"1733988722277006","_RUNTIME_SCOPE":"system","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events"} {"SYSLOG_FACILITY":"3","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","__REALTIME_TIMESTAMP":"1733988723277782","_CAP_EFFECTIVE":"0","_PID":"526","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_EXE":"/usr/bin/cib","_COMM":"cib","_RUNTIME_SCOPE":"system","_UID":"1001","__MONOTONIC_TIMESTAMP":"3277010798","_TRANSPORT":"stdout","_SYSTEMD_UNIT":"radicle-ci-broker.service","_SYSTEMD_SLICE":"system.slice","_SELINUX_CONTEXT":"unconfined\n","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:03.277600Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","PRIORITY":"6","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","SYSLOG_IDENTIFIER":"cib","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_GID":"1001","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31363;b=da4ba18425ea4e34bd0f0731f0270572;m=c353376e;t=6290db5e53fd6;x=f0840763406ccc13","_HOSTNAME":"radicle-ci","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572"} {"_SYSTEMD_UNIT":"radicle-ci-broker.service","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31364;b=da4ba18425ea4e34bd0f0731f0270572;m=c3627c05;t=6290db5f4846c;x=5bded561567c656f","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:04.278174Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_RUNTIME_SCOPE":"system","_HOSTNAME":"radicle-ci","_GID":"1001","PRIORITY":"6","_SELINUX_CONTEXT":"unconfined\n","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","__MONOTONIC_TIMESTAMP":"3278011397","_UID":"1001","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","SYSLOG_IDENTIFIER":"cib","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_TRANSPORT":"stdout","_PID":"526","_COMM":"cib","__REALTIME_TIMESTAMP":"1733988724278380","SYSLOG_FACILITY":"3","_CAP_EFFECTIVE":"0","_SYSTEMD_SLICE":"system.slice","_EXE":"/usr/bin/cib","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e"} {"PRIORITY":"6","SYSLOG_FACILITY":"3","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_TRANSPORT":"stdout","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_GID":"1001","__REALTIME_TIMESTAMP":"1733988725278778","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:05.278657Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_PID":"526","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_SYSTEMD_UNIT":"radicle-ci-broker.service","_EXE":"/usr/bin/cib","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","SYSLOG_IDENTIFIER":"cib","_HOSTNAME":"radicle-ci","_COMM":"cib","_SYSTEMD_SLICE":"system.slice","__MONOTONIC_TIMESTAMP":"3279011794","_RUNTIME_SCOPE":"system","_UID":"1001","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31367;b=da4ba18425ea4e34bd0f0731f0270572;m=c371bfd2;t=6290db603c83a;x=b50947c211c1edcb","_SELINUX_CONTEXT":"unconfined\n","_CAP_EFFECTIVE":"0"} {"SYSLOG_IDENTIFIER":"cib","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_TRANSPORT":"stdout","_PID":"526","PRIORITY":"6","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_GID":"1001","_SYSTEMD_SLICE":"system.slice","__REALTIME_TIMESTAMP":"1733988726279175","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_COMM":"cib","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31368;b=da4ba18425ea4e34bd0f0731f0270572;m=c381039f;t=6290db6130c07;x=7b8169758b14d38c","_SELINUX_CONTEXT":"unconfined\n","_RUNTIME_SCOPE":"system","_EXE":"/usr/bin/cib","_CAP_EFFECTIVE":"0","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:06.279078Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_HOSTNAME":"radicle-ci","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_SYSTEMD_UNIT":"radicle-ci-broker.service","SYSLOG_FACILITY":"3","_UID":"1001","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","__MONOTONIC_TIMESTAMP":"3280012191"} {"MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:07.279440Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_EXE":"/usr/bin/cib","_RUNTIME_SCOPE":"system","_TRANSPORT":"stdout","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_SELINUX_CONTEXT":"unconfined\n","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_PID":"526","_HOSTNAME":"radicle-ci","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","__REALTIME_TIMESTAMP":"1733988727279541","_SYSTEMD_UNIT":"radicle-ci-broker.service","PRIORITY":"6","_GID":"1001","_UID":"1001","SYSLOG_FACILITY":"3","_COMM":"cib","SYSLOG_IDENTIFIER":"cib","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=31369;b=da4ba18425ea4e34bd0f0731f0270572;m=c390474d;t=6290db6224fb5;x=faafb09a153285ff","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","_CAP_EFFECTIVE":"0","_SYSTEMD_SLICE":"system.slice","__MONOTONIC_TIMESTAMP":"3281012557"} {"_PID":"526","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","_HOSTNAME":"radicle-ci","__MONOTONIC_TIMESTAMP":"3282012856","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"cib","_SYSTEMD_SLICE":"system.slice","_COMM":"cib","_CAP_EFFECTIVE":"0","_SELINUX_CONTEXT":"unconfined\n","__REALTIME_TIMESTAMP":"1733988728279841","_SYSTEMD_UNIT":"radicle-ci-broker.service","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_GID":"1001","SYSLOG_FACILITY":"3","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=3136a;b=da4ba18425ea4e34bd0f0731f0270572;m=c39f8ab8;t=6290db6319321;x=d04fd63cc7903199","PRIORITY":"6","_EXE":"/usr/bin/cib","MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:08.279755Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_RUNTIME_SCOPE":"system","_UID":"1001"} {"MESSAGE":"{\"timestamp\":\"2024-12-12T07:32:09.280049Z\",\"level\":\"TRACE\",\"fields\":{\"message\":\"event queue length\",\"msg_id\":\"QueueProcQueueLength\",\"kind\":\"debug\",\"len\":\"0\"}}","_UID":"1001","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_HOSTNAME":"radicle-ci","_PID":"526","_TRANSPORT":"stdout","__REALTIME_TIMESTAMP":"1733988729280146","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","_RUNTIME_SCOPE":"system","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_GID":"1001","PRIORITY":"6","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_CAP_EFFECTIVE":"0","_SYSTEMD_UNIT":"radicle-ci-broker.service","_EXE":"/usr/bin/cib","SYSLOG_IDENTIFIER":"cib","_COMM":"cib","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","_SYSTEMD_SLICE":"system.slice","SYSLOG_FACILITY":"3","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=3136b;b=da4ba18425ea4e34bd0f0731f0270572;m=c3aece29;t=6290db640d692;x=b38aefb25148da02","__MONOTONIC_TIMESTAMP":"3283013161"} {"__MONOTONIC_TIMESTAMP":"1284501864","_PID":"526","__REALTIME_TIMESTAMP":"1733986730768848","_EXE":"/usr/bin/cib","__CURSOR":"s=f30566e7c34e4ba190c6c671ff826507;i=2d8f2;b=da4ba18425ea4e34bd0f0731f0270572;m=4c8ff168;t=6290d3f21f9d0;x=d9b8c7ef2053cc0f","_SYSTEMD_CGROUP":"/system.slice/radicle-ci-broker.service","_CAP_EFFECTIVE":"0","_SYSTEMD_SLICE":"system.slice","_SELINUX_CONTEXT":"unconfined\n","_CMDLINE":"/bin/cib --log-level trace --config /home/_rad/ci-broker.yaml process-events","_UID":"1001","SYSLOG_IDENTIFIER":"cib","_RUNTIME_SCOPE":"system","_BOOT_ID":"da4ba18425ea4e34bd0f0731f0270572","_MACHINE_ID":"35c654cc069c402d8cb0b34b91f12e5e","_SYSTEMD_UNIT":"radicle-ci-broker.service","_SYSTEMD_INVOCATION_ID":"949b57247eb24bb5aefcec93f049beab","SYSLOG_FACILITY":"3","_COMM":"cib","_STREAM_ID":"9c46f4f6f0074e07986a0c3769f6545e","_HOSTNAME":"radicle-ci","_GID":"1001","MESSAGE":"{\"timestamp\":\"2024-12-12T06:58:50.768787Z\",\"level\":\"INFO\",\"fields\":{\"message\":\"Finish CI run\",\"msg_id\":\"BrokerRunEnd\",\"kind\":\"finish_run\",\"run\":\"Run { broker_run_id: RunId { id: \\\"62c45727-a4d8-4a29-9dae-88c6e8b61655\\\" }, adapter_run_id: Some(RunId { id: \\\"fa233c62-51df-4812-865c-7b989915c1f3\\\" }), adapter_info_url: Some(\\\"http://radicle-ci/fa233c62-51df-4812-865c-7b989915c1f3/log.html\\\"), repo_id: RepoId(rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5), repo_name: \\\"heartwood\\\", timestamp: \\\"2024-12-12 06:58:03Z\\\", whence: Branch { name: \\\"master\\\", commit: Oid(d9c76893a144fd787654613f2bfb919613014a71), who: Some(\\\"did:key:z6MkiB8T5cBEQHnrs2MgjMVqvpSVj42X81HjKfFi2XBoMbtr (radicle-ci)\\\") }, state: Finished, result: Some(Failure) }\"},\"span\":{\"broker_run_id\":\"62c45727-a4d8-4a29-9dae-88c6e8b61655\",\"name\":\"execute_ci_run\"},\"spans\":[{\"broker_run_id\":\"62c45727-a4d8-4a29-9dae-88c6e8b61655\",\"name\":\"execute_ci_run\"}]}","_TRANSPORT":"stdout","PRIORITY":"6"} ~~~ # Acceptance criteria for logging The CI broker writes log messages to its standard error output (stderr), which the node operator can capture to a suitable persistent location. The logs are structured: each line is a JSON object. The structured logs are meant to be easier to process by programs, for example to extract information for monitoring, and alerting the node operator about problems. An example log message might look like below (here formatted on multiple lines for human consumption): ~~~json { "msg": "CI broker starts", "level": "INFO", "ts": "2024-08-14T13:38:36.733953135Z", } ~~~ Because logs are crucial for managing a system, we record acceptance criteria for the minimum logging that the CI broker needs to do. ## Logs start and successful end _What:_ `cib` logs a message when it starts and ends. _Why:_ The program starting to run can be important information, for example, to know when it's not running. It's also important to know if the CI broker terminates successfully. _Who:_ `cib-dev`. We verify this by starting `cib` in a mode where it processes any events already in the event queue, and then terminates. We don't add any events, so `cib` just terminates at once. All of this will work, when properly set up. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a directory reports when I run ./env.sh cib --config broker.yaml queued then stderr contains "CibStart" then stderr contains "CibEndSuccess" ~~~ ## Logs termination due to error _What:_ `cib` logs a message when it ends due to an unrecoverable error. _Why:_ It's quite important to know this. Note that a recoverable error does not terminate the CI broker. _Who:_ `cib-dev`. We check this by running the CI broker without a local node. This is an error it can't recover from. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh when I try to run env RAD_HOME=/does/not/exist cib --config broker.yaml queued then stderr contains "CibStart" then stderr contains "CibEndFailure" ~~~ # Acceptance criteria for reports The CI broker creates HTML and JSON reports on a schedule, as well as when CI runs end. The scenarios in this chapter verify that those reports are as wanted. ## Produces a JSON status file _What:_ `cib` produces a JSON status file with information about the current state of the CI broker. _Why:_ This makes it easy to monitor the CI broker using an automated monitoring system. _Who:_ `node-ops` ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a directory reports when I run cib --config broker.yaml queued then file reports/status.json exists when I run jq .event_queue_length reports/status.json then stdout is exactly "0\n" ~~~ # Acceptance criteria for upgrades _What:_ The node operator can safely upgrade the CI broker. At the very least, the CI broker developers need to know if they are making a breaking change. _Why:_ If software upgrades are tedious or risky, they happen less often, to the detriment of everyone. _Who:_ `cib-dev`, `node-ops` It is important that those running the CI broker can upgrade confidently. This requires, at least, that CI broker upgrades in existing installations do not break anything, or at least not without warning. The scenario in this chapter verifies that in a simple, even simplistic manner. Note that this upgrade testing is very much in its infancy. It is expected to be fleshed out over time. There will probably be more scenarios later. The overall approach is as follows: * we run various increasing versions of the CI broker * we use the same configuration file and database for each version * we have an isolated test node so that the CI broker can validate repository and commit * for each version, we use `cibtool trigger` and `cib queued` to run CI * after each version, we verify that the database has all the CI runs it had before running the version, plus one more Note that because this scenario may be run outside the developer's development environment, it is currently difficult to access the Git tags that represent the CI broker releases. Thus we verify upgrades to the Git commit identifiers instead. Note that this should be commits, not tag objects, as the tests may need to run in clone of the a Git repository without tags. This scenario needs to be updated when a new release has been made, to avoid the test suite taking too long to run. The goal is to verify, across releases, that upgrades from each release to the next is verified to work. Thus, given releases 1, 2, 3, etc, we amend the scenario to drop all but latest release, and add any missing release. However, if we've neglected to update the scenario for a release, we make sure we don't break the chain. |release|scenario has| |-|:-| |1|none| |2|1 HEAD| |3|1 2 HEAD| |4|2 3 HEAD| |5|2 3 HEAD| |6|3 4 5 HEAD| |7|5 6 HEAD| Release can't do upgrade tests, but it's long in the past so that's OK. Release 2 upgrades from release 1 to HEAD, the current tip of the branch. Release 3 upgrades from 1 to 2 to HEAD. Release 4 can drop release 1, but adds 3. After release 5 we forgot to update the scenario, so for release 6 we include testing upgrade to release 4. For release 7 we can again trip the list. This doesn't verify that upgrades work if we skip releases. We're OK with that, until users say they want to skip and are having trouble. ~~~scenario given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh given a Git repository testy in the Radicle node given file verify-upgrade given a directory reports when I touch file run-list.txt when I run ./env.sh bash -x verify-upgrade run-list.txt f307ac433d4c100578680be355cac65db2f3f9dc when I run ./env.sh bash -x verify-upgrade run-list.txt HEAD ~~~ ~~~{#verify-upgrade .file .sh} #!/bin/sh # # Given a list of CI runs and a CI broker version, build and run that # version so that it triggers and runs CI on a given change. Then # verify the CI broker database has the CI runs in the list, plus one # more, and then update the list. set -eu REPO="testy" LIST="$1" VERSION="$2" # Unset this so that the Cargo cache doesn't get messed up. (This # smells like a caching bug, or my misundestanding.) unset CARGO_TARGET_DIR # Remember where various things are. db="$(pwd)/ci-broker.db" reports="$(pwd)/reports" adapter="$(pwd)/adapter.sh" # Remember where the config is and update config to use correct # database and report directory. config="$(pwd)/broker.yaml" sed -i "s,^db:.*,db: $db," "$config" sed -i "s,^report_dir:.*,report_dir: $reports," "$config" sed -i "s,command:.*,command: $adapter," "$config" nl "$config" # Get source code for CI broker. The scenario that uses this script # set $SRCDIR to point at the source tree, so we get the source code # from there to avoid having to fetch things from the network. rm -rf ci-broker html mkdir ci-broker html export SRCDIR="$CARGO_MANIFEST_DIR" (cd "$SRCDIR" && git archive "$VERSION") | tar -C ci-broker -xf - # Do things in the exported CI broker source tree. Capture stdout to a # new list of CI run. ( cd ci-broker # Build source code. find -name '*.rs' -exec sed -Ei '/\[deny\(/d' '{}' + cargo build --all-targets (echo "Old CI run lists:" cargo run -q --bin cibtool -- --db "$db" run list 1>&2 cargo run -q --bin cibtool -- --db "$db" run list --json) 1>&2 # Trigger a CI run. Hide the event ID that cibtool writes to # stdout. cargo run -q --bin cibtool -- --db "$db" trigger --repo "$REPO" --ref main --commit HEAD >/dev/null # Run CI on queued events. cargo run -q --bin cib -- --config "$config" queued # List CI runs now in database. cargo run -q --bin cibtool -- --db "$db" run list ) >"$LIST.new" # Check that new list contains everything in old list, plus one more. removed="$(diff -u <(sort "$LIST") <(sort "$LIST.new") | sed '1,/^@@/d' | grep -c "^-" || true)" added="$(diff -u <(sort "$LIST") <(sort "$LIST.new") | sed '1,/^@@/d' | grep -c "^+" || true)" if [ "$removed" = 0 ] && [ "$added" = 1 ]; then echo "CI broker $VERSION ran OK" mv "$LIST.new" "$LIST" else echo "CI broker removed $removed, added $added CI runs." 1>&2 exit 1 fi ~~~