radicle-ci-broker/ci-broker.md

Introduction

This document describes the acceptance criteria for the Radicle CI broker, as well as how to verify that they are met. Acceptance criteria here means a requirement that must be met for the software to be acceptable to its stakeholders.

This file is used by Subplot to generate and run test code as part of running cargo test.

Data files shared between scenarios

Broker configuration

db: ci-broker.db
report_dir: reports
default_adapter: mcadapterface
adapters:
  mcadapterface:
    command: ./adapter.sh
    env:
      RADICLE_NATIVE_CI: native-ci.yaml
    sensitive_env: {}
filters:
  - !Branch "main"

db: ci-broker.db
report_dir: reports
default_adapter: mcadapterface
adapters:
  mcadapterface:
    command: ./adapter.sh
    env:
      RADICLE_NATIVE_CI: native-ci.yaml
    sensitive_env: {}
filters:
  - !Branch "this-branch-does-not-exist"

A dummy adapter

This adapter does nothing, just reports a run ID and a successful run.

#!/bin/bash
set -euo pipefail
echo '{"response":"triggered","run_id":{"id":"xyzzy"}}'
echo '{"response":"finished","result":"success"}'

A refsFetched node event

This is a node event from the node to signal that some git refs have changed in repository.

{
  "type": "refsFetched",
  "remote": "z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV",
  "rid": "rad:zwTxygwuz5LDGBq255RA2CbNGrz8",
  "updated": [
    {
      "updated": {
        "name": "refs/namespaces/DUMMYNID/refs/heads/main",
        "old": "0000000000000000000000000000000000000000",
        "new": "0000000000000000000000000000000000000000"
      }
    }
  ]
}

A shutdown event

This asks the CI broker to shut down cleanly.

{
  "type": "refsFetched",
  "remote": "z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV",
  "rid": "rad:zwTxygwuz5LDGBq255RA2CbNGrz8",
  "updated": [
    {
      "skipped": {
        "name": "shutdown",
        "oid": "0000000000000000000000000000000000000000"
      }
    }
  ]
}

Set rid in refsFetched event

This is a helper script that reads a refsFetched event and changes its rid field to be the id of the given repository. It also sets the name field for updated refs to include the repository ID. It writes the message back to its file.

#!/usr/bin/python3

import json, sys
from subprocess import run, PIPE

filename = sys.argv[1]
cwd = sys.argv[2]

p = run(["rad", "."], check=True, capture_output=True, cwd=cwd)
rid = p.stdout.decode().strip()

p = run(["rad", "self", "--nid"], check=True, capture_output=True, cwd=cwd)
nid = p.stdout.decode().strip()

p = run(["git", "rev-parse", "HEAD"], check=True, capture_output=True, cwd=cwd)
oid = p.stdout.decode().strip()

o = json.load(open(filename))

o["rid"] = rid

if "updated" in o:
    x = o["updated"]
    for oo in x:
        name = oo["updated"]["name"]
        oo["updated"]["name"] = nid.join(name.split("DUMMYNID"))
        oo["updated"]["new"] = oid
    o["updated"] = x

with open(filename, "w") as f:
    json.dump(o, fp=f, indent=4)

Set environment variables and run command

To avoid having to repeat environment variables to set up and use a Radicle node for verification scenarios, we provide a script that sets them and runs a command.

#!/bin/bash

set -euo pipefail

homedir="$(pwd)/homedir"

env \
	HOME="$homedir" \
	RAD_PASSPHRASE=secret \
	RAD_HOME="$homedir/.radicle" \
	RAD_SOCKET=synt.sock \
	RUST_LOG=trace \
	RADICLE_CI_BROKER_LOG=trace \
	"$@"

Set up a node with repository

Most of our verification scenarios will need to set up a Radicle node and a test repository there. Rather than repeat it in every scenario, we use this helper script.

#!/bin/bash

set -xeuo pipefail

mkdir -p "$HOME"

env | grep RAD
rad auth --alias brokertest

git config --global user.email radicle@example.com
git config --global user.name TestyMcTestFace
git init -b main testy

cd testy
echo "test file" > test.txt

git add .
git commit -am test
git status
git show
rad init --name testy --description test --default-branch main --private --no-confirm --no-seed
rad inspect --identity
rad id list

Acceptance criteria

Shows config as JSON

Requirement: The CI broker can write out the configuration is uses at run time as JSON.

Justification: This is helpful for the node operator to verify that they have configured the program correctly.

Stakeholder: Lars

Our verification here is quite simplistic, and only checks that the output is in the JSON format. It does not try to make sure the JSON matches the YAML semantically.

given an installed cib
given file broker.yaml
when I run cib --config broker.yaml config --output actual.json
when I run jq . actual.json
then command is successful

Smoke test: Runs adapter

Requirement: CI broker can run its adapter.

Justification: This is obviously necessary. If this doesn't work, nothing else has a hope of working.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed synthetic-events
given file refsfetched.json
given file shutdown.json
given file set-rid
when I run env HOME=../homedir python3 set-rid refsfetched.json testy
when I run synthetic-events synt.sock refsfetched.json shutdown.json --log log.txt

given an installed cib
given a directory reports
given file broker.yaml
given file adapter.sh from dummy.sh
when I run chmod +x adapter.sh

when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert
then command is successful
when I run cibtool --db ci-broker.db event shutdown

given an installed cibtool
when I run cibtool --db ci-broker.db event list
when I run cibtool --db ci-broker.db run list

when I try to run bash radenv.sh cib --config broker.yaml queued
then command is successful

when I run cibtool --db ci-broker.db run list
then stdout contains "id: "xyzzy""

Adapter can provide URL for info on run

Requirement: The adapter can provide a URL for information about the run, such a run log. This optional.

Justification: The CI broker does not itself store the run log, but it's useful to be able to point users at one. The CI broker can put that into a Radicle COB or otherwise store it so that users can see it. Note, however, that the adapter gets to decide which URL to provide: it need not be the run log. It might, for example, be a URL to the web view of a "pipeline" in GitLab CI instead, from which the user can access individual logs.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed synthetic-events
given file refsfetched.json
given file shutdown.json
given file set-rid
when I run env HOME=../homedir python3 set-rid refsfetched.json testy
when I run synthetic-events synt.sock refsfetched.json shutdown.json --log log.txt

given file adapter.sh from adapter-with-url.sh
when I run chmod +x adapter.sh

given an installed cib
given file broker.yaml
given a directory reports

when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert
then command is successful

#!/bin/bash
set -euo pipefail
echo '{"response":"triggered","run_id":{"id":"xyzzy"},"info_url":"https://ci.example.com/xyzzy"}'
echo '{"response":"finished","result":"success"}'

Gives helpful error message if node socket can't be found

Requirement: If the CI broker can't connect to the Radicle node control socket, it gives an error message that helps the user to understand the problem.

Justification: This helps users deal with problems themselves and reduces the support burden on the Radicle project.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given file broker.yaml
when I try to run bash radenv.sh RAD_SOCKET=xyzzy.sock cib --config broker.yaml insert
then command fails
then stderr contains "node control socket does not exist: xyzzy.sock"

Gives helpful error message if it doesn't understand its configuration file

Requirement: If the CI broker is given a configuration file that it can't understand, it gives an error message that explains the problem to the user.

Justification: This helps users deal with problems themselves and reduces the support burden on the Radicle project.

Stakeholder: Lars.

Comment: This is a very basic scenario. Error handling is by nature a thing that can always be made better. We can later add more scenarios if we tighten the acceptance criteria.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given file broker.yaml
given file not-yaml.yaml
when I try to run env HOME=homedir cib --config not-yaml.yaml config
then command fails
then stderr contains "failed to parse configuration file as YAML: not-yaml.yaml"

This file is not YAML.

Stops if the node connection breaks

Requirement: If the connection to the Radicle node, via its control socket, breaks, the CI broker terminates with a message saying why.

Justification: The CI broker can either keep running and trying to re-connect, or it can terminate. Either is workable. However, it's a simpler design and less code to terminate and allow re-starting to be handled by a dedicated system, such as systemd.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given an installed synthetic-events
when I run synthetic-events synt.sock --log log.txt
given file broker.yaml
when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert
then stderr contains "connection to the node control socket broke"

Shuts down when requested

Requirement: The test suite can request the CI broker to shut down cleanly, and it doesn't result in an error.

Justification: In the integration test suite, we need to start and stop the CI broker many times. We need to easily detect errors.

Stakeholder: Lars.

We use a special magic fake node event to signal shutdown: a RefsFetched event with a skipped update for a ref "shutdown" and an object id of all zeros. This should be sufficiently impossible to happen in real life.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given an installed synthetic-events
given file shutdown.json
given file broker.yaml
when I run synthetic-events synt.sock shutdown.json --log synt.log
when I try to run bash radenv.sh RAD_SOCKET=synt.sock cib --config broker.yaml insert
then command is successful

Acceptance criteria for test tooling

The event synthesizer is a helper to feed the CI broker node events in a controlled fashion.

Dummy adapter runs successfully

Requirement: The dummy adapter (in embedded file dummy.sh) runs successfully.

Justification: Test scenarios using the dummy adapter need to be able to rely that it works.

Stakeholder: Lars

given file dummy.sh
when I run chmod +x dummy.sh
when I try to run ./dummy.sh
then command is successful

Adapter with URL runs successfully

Requirement: The adapter with a URL (in embedded file adapter-with-url.sh) runs successfully.

Justification: Test scenarios using this adapter need to be able to rely that it works.

Stakeholder: Lars

given file adapter-with-url.sh
when I run chmod +x adapter-with-url.sh
when I try to run ./adapter-with-url.sh
then command is successful

Event synthesizer terminates after first connection

Requirement: The event synthesizer runs in the background, but terminates after the first connection.

Justification: This is needed so that it can be invoked in Subplot scenarios.

Stakeholder: Lars.

The following scenario may only work on Linux, as it's using pgrep and nc and those may not be portable. If so, this may need to be changed for other platforms.

Note also that the version of nc must support the -U option, which in Debian means using the netcat-openbsd package.

We sleep for a very short time to make sure the synthetic-events daemon has time to remove the socket file before we check that it's been deleted.

given an installed synthetic-events

then file synt.sock does not exist

when I run synthetic-events synt.sock
then file synt.sock exists

when I run nc -U synt.sock
when I run sleep 0.1
then file synt.sock does not exist

Acceptance criteria for persistent database

The CI broker uses an SQLite database for persistent data. Many processes may need to access or modify the database at the same time. While SQLite is good at managing that, it needs to be used in the right way for everything to work correctly. The acceptance criteria in this chapter address that.

To enable the verification of these acceptance criteria, the CI broker database allows for a "counter", as a single row in a dedicated table. Concurrency is tested by having multiple processes update the counter at the same time and verifying the end result is as intended and that every value is set exactly once.

Count in a single process

Requirement: A single process can increment the test counter correctly.

Justification: If this doesn't work with a single process, it won't work of multiple processes, either.

Stakeholder: Lars.

given an installed cibtool
then file count.db does not exist
when I run cibtool --db count.db counter show
then stdout is exactly "0\n"
when I run cibtool --db count.db counter count --goal 1000
when I run cibtool --db count.db counter show
then stdout is exactly "1000\n"

Insert events into queue

Requirement: Insert broker events generated from node events into persistent event queue in the database, when allowed by the CI broker event filter.

Justification: This is fundamental for running CI when repositories in a node change.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given an installed synthetic-events

given file refsfetched.json
given file shutdown.json
when I run synthetic-events synt.sock refsfetched.json shutdown.json --log synt.log

given file broker.yaml
when I try to run bash radenv.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert
then command is successful

when I run cibtool --db ci-broker.db event list --verbose
then stdout contains "RefChanged"
then stdout contains "oid: Oid(0000000000000000000000000000000000000000)"
then stdout contains "old: Some(Oid(0000000000000000000000000000000000000000))"

Insert many events into queue

Requirement: Insert many events that arrive quickly.

Justification: We need at least some rudimentary performance testing.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given an installed synthetic-events

given file refsfetched.json
given file shutdown.json
when I run synthetic-events synt.sock refsfetched.json shutdown.json --log synt.log --repeat 1000

given file broker.yaml
when I try to run bash radenv.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert
then command is successful


when I run cibtool --db ci-broker.db event count
then stdout is exactly "1000\n"

Don't insert events into queue when not allowed by filter

Requirement: Nothing is inserted into the persistent event queue then the CI broker's filter does not allow any events.

Justification: This is fundamental for running CI when repositories in a node change.

Stakeholder: Lars.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cib
given an installed synthetic-events

given file refsfetched.json
given file shutdown.json
when I run synthetic-events synt.sock refsfetched.json shutdown.json --log synt.log

given file broker.yaml from broker-allow-nothing.yaml
when I try to run bash radenv.sh env RAD_SOCKET=synt.sock cib --config broker.yaml insert
then command is successful

when I run cibtool --db ci-broker.db event count
then stdout is exactly "0\n"

Process queued events

Requirement: It's possible to run the CI broker in a mode where it only processes events from its persistent event queue.

Justification: This is primarily useful for testing the CI broker queuing implementation.

Stakeholders: Lars.

We verify this by adding events to the queue with cibtool, and then running the CI broker and verifying it terminates after processing the events. We carefully add a shutdown event so that the CI broker shuts down.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given file adapter.sh from dummy.sh
when I run chmod +x adapter.sh

given an installed cib
given an installed cibtool

when I run bash radenv.sh cibtool --db ci-broker.db event add --repo testy --ref main --commit HEAD
when I run cibtool --db ci-broker.db event shutdown

given file broker.yaml
when I run ls -l adapter.sh
when I run bash radenv.sh cib --config broker.yaml queued
then stderr contains "Action: run:"
then stderr contains "Action: shutdown"

when I run cibtool --db ci-broker.db event list
then stdout is exactly ""

when I run cibtool --db ci-broker.db run list
then stdout contains "Success"

Count in concurrent processes

Requirement: Two process can concurrently increment the test counter correctly.

Justification: This is necessary, if not necessarily sufficient, for concurrent database use to work correctly.

Stakeholder: Lars.

Due to limitations in Subplot we mange the concurrent processes using a helper shell script,k count.sh, found below. It runs two concurrent cibtool processes that update the same database file, and count to a desired goal. The script then verifies that everything went correctly.

given an installed cibtool
given file count.sh
when I run bash -x count.sh 100 10
then stdout contains "OK\n"

#!/bin/bash

set -euo pipefail

run() {
	cibtool --db "$DB" counter count --goal "$goal"
}

DB=count.db

goal="$1"
reps="$2"

for x in $(seq "$reps"); do
	echo "Repetition $x"

	rm -f "$DB" ./?.out

	run >1.out 2>&1 &
	one=$!

	run >2.out 2>&1 &
	two=$!

	if ! wait "$one"; then
		echo "first run failed"
		cat 1.out
		exit 1
	fi

	if ! wait "$two"; then
		echo "second run failed"
		cat 2.out
		exit 1
	fi

	if grep ERROR ./?.out; then
		echo found ERRORs
		exit 1
	fi

	n="$(sqlite3 "$DB" 'select counter from counter_test')"
	[ "$n" == "$goal" ] || (
		echo "wrong count $n"
		exit 1
	)

	if awk '/increment to/ { print $NF }' ./?.out | sort -n | uniq -d | grep .; then
		echo "duplicate increments"
		exit 1
	fi
done

echo OK

Acceptance criteria for event queue management

The CI builder database contains a queue of broker events. The cibtool management tool can be used to examine and manipulate the queue.

Events can be queued and removed from queue

Requirement: cibtool can show the queued events, can inject an event, and remove an event.

Justification: This is the minimum functionality needed to manage the event queue.

Stakeholder: Lars.

We verify that this works by adding a new broker event, and then removing it. We randomly choose the repository id for the CI broker itself for this test, but the id shouldn't matter, it just needs to be of the correct form.

given file radenv.sh
given file setup-node.sh
when I run bash radenv.sh bash setup-node.sh

given an installed cibtool
when I run cibtool --db x.db event list
then stdout is exactly ""

when I run bash radenv.sh cibtool --db x.db event add --repo testy --ref main --commit HEAD --base c0ffee --id-file id.txt

when I run cibtool --db x.db event show --id-file id.txt
then stdout contains "rad:"
then stdout contains "main"
then stdout contains "c0ffee"

when I run cibtool --db x.db event remove --id-file id.txt

when I run cibtool --db x.db event list
then stdout is exactly ""

Can add shutdown event to queue

Requirement: cibtool can add a shutdown event to the queued events.

Justification: This is needed for testing, and for the node operator to be able to do this cleanly.

given an installed cibtool
when I run cibtool --db x.db event list
then stdout is exactly ""

when I run cibtool --db x.db event shutdown --id-file id.txt

when I run cibtool --db x.db event show --id-file id.txt
then stdout contains "Shutdown"