# Introduction This document describes the acceptance criteria for the Radicle CI broker, as well as how to verify that they are met. Acceptance criteria here means a requirement that must be met for the software to be acceptable to its stakeholders. This file is used by [Subplot](https://subplot.tech/) to generate and run test code as part of running `cargo test`. # Data files shared between scenarios ## Broker configuration ~~~{#broker.yaml .file .yaml} db: ci-broker.db report_dir: reports default_adapter: mcadapterface adapters: mcadapterface: command: ./adapter.sh env: RADICLE_NATIVE_CI: native-ci.yaml sensitive_env: {} filters: - !Branch "main" ~~~ ~~~{#broker-allow-nothing.yaml .file .yaml} db: ci-broker.db report_dir: reports default_adapter: mcadapterface adapters: mcadapterface: command: ./adapter.sh env: RADICLE_NATIVE_CI: native-ci.yaml sensitive_env: {} filters: - !Branch "this-branch-does-not-exist" ~~~ ## A dummy adapter This adapter does nothing, just reports a run ID and a successful run. ~~~{#dummy.sh .file .sh} #!/bin/bash set -euo pipefail echo '{"response":"triggered","run_id":{"id":"xyzzy"}}' echo '{"response":"finished","result":"success"}' ~~~ ## A `refsFetched` node event This is a node event from the node to signal that some git refs have changed in repository. ~~~{#refsfetched.json .file .json} { "type": "refsFetched", "remote": "z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV", "rid": "rad:zwTxygwuz5LDGBq255RA2CbNGrz8", "updated": [ { "updated": { "name": "refs/namespaces/DUMMYNID/refs/heads/main", "old": "0000000000000000000000000000000000000000", "new": "0000000000000000000000000000000000000000" } } ] } ~~~ ## A shutdown event This asks the CI broker to shut down cleanly. ~~~{#shutdown.json .file .json} { "type": "refsFetched", "remote": "z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV", "rid": "rad:zwTxygwuz5LDGBq255RA2CbNGrz8", "updated": [ { "skipped": { "name": "shutdown", "oid": "0000000000000000000000000000000000000000" } } ] } ~~~ ## Set rid in `refsFetched` event This is a helper script that reads a `refsFetched` event and changes its `rid` field to be the id of the given repository. It also sets the `name` field for updated refs to include the repository ID. It writes the message back to its file. ~~~{#set-rid .file .python} #!/usr/bin/python3 import json, sys from subprocess import run, PIPE filename = sys.argv[1] cwd = sys.argv[2] p = run(["rad", "."], check=True, capture_output=True, cwd=cwd) rid = p.stdout.decode().strip() p = run(["rad", "self", "--nid"], check=True, capture_output=True, cwd=cwd) nid = p.stdout.decode().strip() p = run(["git", "rev-parse", "HEAD"], check=True, capture_output=True, cwd=cwd) oid = p.stdout.decode().strip() o = json.load(open(filename)) o["rid"] = rid if "updated" in o: x = o["updated"] for oo in x: name = oo["updated"]["name"] oo["updated"]["name"] = nid.join(name.split("DUMMYNID")) oo["updated"]["new"] = oid o["updated"] = x with open(filename, "w") as f: json.dump(o, fp=f, indent=4) ~~~ ## Set environment variables and run command To avoid having to repeat environment variables to set up and use a Radicle node for verification scenarios, we provide a script that sets them and runs a command. ~~~{#radenv.sh .file .sh} #!/bin/bash set -euo pipefail homedir="$(pwd)/homedir" env \ HOME="$homedir" \ RAD_PASSPHRASE=secret \ RAD_HOME="$homedir/.radicle" \ RAD_SOCKET=synt.sock \ RUST_LOG=trace \ RADICLE_CI_BROKER_LOG=trace \ "$@" ~~~ ## Set up a node with repository Most of our verification scenarios will need to set up a Radicle node and a test repository there. Rather than repeat it in every scenario, we use this helper script. ~~~{#setup-node.sh .file .sh} #!/bin/bash set -xeuo pipefail mkdir -p "$HOME" env | grep RAD rad auth --alias brokertest git config --global user.email radicle@example.com git config --global user.name TestyMcTestFace git init -b main testy cd testy echo "test file" > test.txt git add . git commit -am test git status git show rad init --name testy --description test --default-branch main --private --no-confirm --no-seed rad inspect --identity rad id list ~~~ # Acceptance criteria ## Shows config as JSON _Requirement:_ The CI broker can write out the configuration is uses at run time as JSON. _Justification:_ This is helpful for the node operator to verify that they have configured the program correctly. _Stakeholder:_ Lars Our verification here is quite simplistic, and only checks that the output is in the JSON format. It does not try to make sure the JSON matches the YAML semantically. ~~~scenario given an installed cib given file broker.yaml when I run cib --config broker.yaml config --output actual.json when I run jq . actual.json then command is successful ~~~ ## Smoke test: Runs adapter _Requirement:_ CI broker can run its adapter. _Justification:_ This is obviously necessary. If this doesn't work, nothing else has a hope of working. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed synthetic-events given file refsfetched.json given file shutdown.json given file set-rid when I run env HOME=../homedir python3 set-rid refsfetched.json testy when I run synthetic-events synt.sock refsfetched.json shutdown.json --log log.txt given an installed cib given a directory reports given file broker.yaml given file adapter.sh from dummy.sh when I run chmod +x adapter.sh when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful when I run cibtool --db ci-broker.db event shutdown given an installed cibtool when I run cibtool --db ci-broker.db event list when I run cibtool --db ci-broker.db run list when I try to run bash radenv.sh cib --config broker.yaml queued then command is successful when I run cibtool --db ci-broker.db run list then stdout contains "id: "xyzzy"" ~~~ ## Adapter can provide URL for info on run _Requirement:_ The adapter can provide a URL for information about the run, such a run log. This optional. _Justification:_ The CI broker does not itself store the run log, but it's useful to be able to point users at one. The CI broker can put that into a Radicle COB or otherwise store it so that users can see it. Note, however, that the adapter gets to decide which URL to provide: it need not be the run log. It might, for example, be a URL to the web view of a "pipeline" in GitLab CI instead, from which the user can access individual logs. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed synthetic-events given file refsfetched.json given file shutdown.json given file set-rid when I run env HOME=../homedir python3 set-rid refsfetched.json testy when I run synthetic-events synt.sock refsfetched.json shutdown.json --log log.txt given file adapter.sh from adapter-with-url.sh when I run chmod +x adapter.sh given an installed cib given file broker.yaml given a directory reports when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful ~~~ ~~~{#adapter-with-url.sh .file .sh} #!/bin/bash set -euo pipefail echo '{"response":"triggered","run_id":{"id":"xyzzy"},"info_url":"https://ci.example.com/xyzzy"}' echo '{"response":"finished","result":"success"}' ~~~ ## Gives helpful error message if node socket can't be found _Requirement:_ If the CI broker can't connect to the Radicle node control socket, it gives an error message that helps the user to understand the problem. _Justification:_ This helps users deal with problems themselves and reduces the support burden on the Radicle project. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given file broker.yaml when I try to run bash radenv.sh RAD_SOCKET=xyzzy.sock cib --config broker.yaml insert then command fails then stderr contains "node control socket does not exist: xyzzy.sock" ~~~ ## Gives helpful error message if it doesn't understand its configuration file _Requirement:_ If the CI broker is given a configuration file that it can't understand, it gives an error message that explains the problem to the user. _Justification:_ This helps users deal with problems themselves and reduces the support burden on the Radicle project. _Stakeholder:_ Lars. _Comment:_ This is a very basic scenario. Error handling is by nature a thing that can always be made better. We can later add more scenarios if we tighten the acceptance criteria. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given file broker.yaml given file not-yaml.yaml when I try to run env HOME=homedir cib --config not-yaml.yaml config then command fails then stderr contains "failed to parse configuration file as YAML: not-yaml.yaml" ~~~ ~~~{#not-yaml.yaml .file} This file is not YAML. ~~~ ## Stops if the node connection breaks _Requirement:_ If the connection to the Radicle node, via its control socket, breaks, the CI broker terminates with a message saying why. _Justification:_ The CI broker can either keep running and trying to re-connect, or it can terminate. Either is workable. However, it's a simpler design and less code to terminate and allow re-starting to be handled by a dedicated system, such as `systemd`. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given an installed synthetic-events when I run synthetic-events synt.sock --log log.txt given file broker.yaml when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert then stderr contains "connection to the node control socket broke" ~~~ ## Shuts down when requested _Requirement:_ The test suite can request the CI broker to shut down cleanly, and it doesn't result in an error. _Justification:_ In the integration test suite, we need to start and stop the CI broker many times. We need to easily detect errors. _Stakeholder:_ Lars. We use a special magic fake node event to signal shutdown: a `RefsFetched` event with a skipped update for a ref "`shutdown`" and an object id of all zeros. This should be sufficiently impossible to happen in real life. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given an installed synthetic-events given file shutdown.json given file broker.yaml when I run synthetic-events synt.sock shutdown.json --log synt.log when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful ~~~ # Acceptance criteria for test tooling The event synthesizer is a helper to feed the CI broker node events in a controlled fashion. ## Dummy adapter runs successfully _Requirement:_ The dummy adapter (in embedded file `dummy.sh`) runs successfully. _Justification:_ Test scenarios using the dummy adapter need to be able to rely that it works. _Stakeholder:_ Lars ~~~scenario given file dummy.sh when I run chmod +x dummy.sh when I try to run ./dummy.sh then command is successful ~~~ ## Adapter with URL runs successfully _Requirement:_ The adapter with a URL (in embedded file `adapter-with-url.sh`) runs successfully. _Justification:_ Test scenarios using this adapter need to be able to rely that it works. _Stakeholder:_ Lars ~~~scenario given file adapter-with-url.sh when I run chmod +x adapter-with-url.sh when I try to run ./adapter-with-url.sh then command is successful ~~~ ## Event synthesizer terminates after first connection _Requirement:_ The event synthesizer runs in the background, but terminates after the first connection. _Justification:_ This is needed so that it can be invoked in Subplot scenarios. _Stakeholder:_ Lars. The following scenario may only work on Linux, as it's using `pgrep` and `nc` and those may not be portable. If so, this may need to be changed for other platforms. Note also that the version of `nc` must support the `-U` option, which in Debian means using the `netcat-openbsd` package. We sleep for a very short time to make sure the `synthetic-events` daemon has time to remove the socket file before we check that it's been deleted. ~~~scenario given an installed synthetic-events then file synt.sock does not exist when I run synthetic-events synt.sock then file synt.sock exists when I run nc -U synt.sock when I run sleep 0.1 then file synt.sock does not exist ~~~ # Acceptance criteria for persistent database The CI broker uses an SQLite database for persistent data. Many processes may need to access or modify the database at the same time. While SQLite is good at managing that, it needs to be used in the right way for everything to work correctly. The acceptance criteria in this chapter address that. To enable the verification of these acceptance criteria, the CI broker database allows for a "counter", as a single row in a dedicated table. Concurrency is tested by having multiple processes update the counter at the same time and verifying the end result is as intended and that every value is set exactly once. ## Count in a single process _Requirement:_ A single process can increment the test counter correctly. _Justification:_ If this doesn't work with a single process, it won't work of multiple processes, either. _Stakeholder:_ Lars. ~~~scenario given an installed cibtool then file count.db does not exist when I run cibtool --db count.db counter show then stdout is exactly "0\n" when I run cibtool --db count.db counter count --goal 1000 when I run cibtool --db count.db counter show then stdout is exactly "1000\n" ~~~ ## Insert events into queue _Requirement:_ Insert broker events generated from node events into persistent event queue in the database, when allowed by the CI broker event filter. _Justification:_ This is fundamental for running CI when repositories in a node change. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given an installed synthetic-events given file refsfetched.json given file shutdown.json when I run synthetic-events synt.sock refsfetched.json shutdown.json --log synt.log given file broker.yaml when I try to run bash radenv.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful when I run cibtool --db ci-broker.db event list --verbose then stdout contains "RefChanged" then stdout contains "oid: Oid(0000000000000000000000000000000000000000)" then stdout contains "old: Some(Oid(0000000000000000000000000000000000000000))" ~~~ ## Insert many events into queue _Requirement:_ Insert many events that arrive quickly. _Justification:_ We need at least some rudimentary performance testing. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given an installed synthetic-events given file refsfetched.json given file shutdown.json when I run synthetic-events synt.sock refsfetched.json shutdown.json --log synt.log --repeat 1000 given file broker.yaml when I try to run bash radenv.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful when I run cibtool --db ci-broker.db event count then stdout is exactly "1000\n" ~~~ ## Don't insert events into queue when not allowed by filter _Requirement:_ Nothing is inserted into the persistent event queue then the CI broker's filter does not allow any events. _Justification:_ This is fundamental for running CI when repositories in a node change. _Stakeholder:_ Lars. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cib given an installed synthetic-events given file refsfetched.json given file shutdown.json when I run synthetic-events synt.sock refsfetched.json shutdown.json --log synt.log given file broker.yaml from broker-allow-nothing.yaml when I try to run bash radenv.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert then command is successful when I run cibtool --db ci-broker.db event count then stdout is exactly "0\n" ~~~ ## Process queued events _Requirement:_ It's possible to run the CI broker in a mode where it only processes events from its persistent event queue. _Justification:_ This is primarily useful for testing the CI broker queuing implementation. _Stakeholders:_ Lars. We verify this by adding events to the queue with `cibtool`, and then running the CI broker and verifying it terminates after processing the events. We carefully add a shutdown event so that the CI broker shuts down. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given file adapter.sh from dummy.sh when I run chmod +x adapter.sh given an installed cib given an installed cibtool when I run bash radenv.sh cibtool --db ci-broker.db event add --repo testy --ref main --commit HEAD when I run cibtool --db ci-broker.db event shutdown given file broker.yaml when I run ls -l adapter.sh when I run bash radenv.sh cib --config broker.yaml queued then stderr contains "Action: run:" then stderr contains "Action: shutdown" when I run cibtool --db ci-broker.db event list then stdout is exactly "" when I run cibtool --db ci-broker.db run list then stdout contains "Success" ~~~ ## Count in concurrent processes _Requirement:_ Two process can concurrently increment the test counter correctly. _Justification:_ This is necessary, if not necessarily sufficient, for concurrent database use to work correctly. _Stakeholder:_ Lars. Due to limitations in Subplot we mange the concurrent processes using a helper shell script,k `count.sh`, found below. It runs two concurrent `cibtool` processes that update the same database file, and count to a desired goal. The script then verifies that everything went correctly. ~~~scenario given an installed cibtool given file count.sh when I run bash -x count.sh 100 10 then stdout contains "OK\n" ~~~ ~~~{#count.sh .file .sh} #!/bin/bash set -euo pipefail run() { cibtool --db "$DB" counter count --goal "$goal" } DB=count.db goal="$1" reps="$2" for x in $(seq "$reps"); do echo "Repetition $x" rm -f "$DB" ./?.out run >1.out 2>&1 & one=$! run >2.out 2>&1 & two=$! if ! wait "$one"; then echo "first run failed" cat 1.out exit 1 fi if ! wait "$two"; then echo "second run failed" cat 2.out exit 1 fi if grep ERROR ./?.out; then echo found ERRORs exit 1 fi n="$(sqlite3 "$DB" 'select counter from counter_test')" [ "$n" == "$goal" ] || ( echo "wrong count $n" exit 1 ) if awk '/increment to/ { print $NF }' ./?.out | sort -n | uniq -d | grep .; then echo "duplicate increments" exit 1 fi done echo OK ~~~ # Acceptance criteria for event queue management The CI builder database contains a queue of broker events. The `cibtool` management tool can be used to examine and manipulate the queue. ## Events can be queued and removed from queue _Requirement:_ `cibtool` can show the queued events, can inject an event, and remove an event. _Justification:_ This is the minimum functionality needed to manage the event queue. _Stakeholder:_ Lars. We verify that this works by adding a new broker event, and then removing it. We randomly choose the repository id for the CI broker itself for this test, but the id shouldn't matter, it just needs to be of the correct form. ~~~scenario given file radenv.sh given file setup-node.sh when I run bash radenv.sh bash setup-node.sh given an installed cibtool when I run cibtool --db x.db event list then stdout is exactly "" when I run bash radenv.sh cibtool --db x.db event add --repo testy --ref main --commit HEAD --base c0ffee --id-file id.txt when I run cibtool --db x.db event show --id-file id.txt then stdout contains "rad:" then stdout contains "main" then stdout contains "c0ffee" when I run cibtool --db x.db event remove --id-file id.txt when I run cibtool --db x.db event list then stdout is exactly "" ~~~ ## Can add shutdown event to queue _Requirement:_ `cibtool` can add a shutdown event to the queued events. _Justification:_ This is needed for testing, and for the node operator to be able to do this cleanly. ~~~scenario given an installed cibtool when I run cibtool --db x.db event list then stdout is exactly "" when I run cibtool --db x.db event shutdown --id-file id.txt when I run cibtool --db x.db event show --id-file id.txt then stdout contains "Shutdown" ~~~