radicle-ci-broker/doc/userguide.md

# Introduction

The Radicle CI broker runs CI for repositories in the local Radicle
node. This is the user guide for the CI broker.

The CI broker helps users run validation on changes to their software
project, by automating the building and testing of the projects when
anything in the repository changes. This is often called "continuous
integration".

(Technically, "continuous integration" is the software development
practice to merge changes into the main line of development
frequently, at least daily, to avoid painful merge conflict
resolutions. However, for this guide we say "CI" to mean "when
repository changes, perform these actions", which is a more generic,
and quite popular definition, if not very purist.)

# Overview

The Radicle node stores Git repositories and synchronizes them with
other Radicle nodes. The CI broker connects to its local node and gets
"node events" whenever anything changes in the node. The relevant
change for the CI broker is that Git references ("refs") in a
repository have been created, updated, or deleted. For now, these are
branches. Later, Radicle and the CI broker will support other
references, such as tags.

There are no node events for Git repositories being created or
deleted. It's not possible to create a Radicle repository without
creating a branch, so just looking at references is enough.

The CI broker looks at the reference changes and refines them into "CI
events", which are more suitable for the kind of CI use that the CI
broker is meant to enable. That is, it does not care about all
"this ref changed" events, which are quite low level.

The CI events are filtered, and events that are allowed by the filter
trigger a CI run, which runs another program called the CI adapter.
The adapter arranges for a CI system or CI engine to execute CI for
the code change captured in the CI event. There are adapters for
different CI systems, such as GitHub Actions, Concourse, Kraken, and
also the "native adapter", that runs a shell script locally on the
host where the adapter runs. Each adapter may require a different
configuration to suit the CI system it targets.

# Getting started

To use Radicle CI, you first need to have a Radicle node where you
want to run CI. Due to technical limitations, this can't be your usual
node, because it won't react to changes you push to it. It will only react to
changes it receives from other nodes.

You can set a passphrase on your node key. If you do, you need to arrange for
the CI broker to know it, by setting `RAD_PASSPHRASE` in the environment, when
you start the CI broker process.

Once you've [installed a Radicle node and it is
running](https://radicle.dev/#get-started):

* Install the Radicle CI broker. One way to do that is by
  building the latest release from source. This requires the Rust
  compiler and tools to be installed.

~~~sh
cargo install radicle-ci-broker --locked
~~~

* Install the Radicle native CI adapter. This is the simplest adapter
  to get to work, even if you don't want to use it later.

~~~sh
cargo install radicle-native-ci --locked
~~~

* Create a configuration file. You can call the file anything you like
  (the example below assumes `ci-broker.yaml`). The example below
  assumes `_rad` user; adjust paths as necessary. You should make sure
  the `report_dir` field points to a directory that exists.

  **IMPORTANT**: You should ensure the Radicle node below (identified
  by the node id after `!Node`) is a node you trust, otherwise you
  could end up running arbitrary code through a random patch created
  by anyone. If your run several nodes, you can list them all within
  an `!Or` expression.

~~~yaml
db: /home/_rad/ci-broker.db
report_dir: /srv/http
queue_len_interval: 1min
adapters:
  native:
    command: /bin/radicle-native-ci
    env:
      RADICLE_NATIVE_CI: /home/_rad/native-ci.yaml
    sensitive_env: {}
triggers:
  - adapter: native
    filters:
    - !And
      - !HasFile ".radicle/native.yaml"
      - !Node z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV
      - !Or
        - !DefaultBranch
        - !PatchCreated
        - !PatchUpdated
~~~

* Create a configuration file for the native CI adapter. Place it in
  the location specified in `ci-broker.yaml` above in the
  `RADICLE_NATIVE_CI` environment variable.

~~~yaml
state: /srv/http
log: /home/_rad/native-ci.log
base_url: http://setup-ci/
~~~

* Start the CI adapter:

~~~sh
cib --config ci-broker.yaml process-events
~~~

You can also set up a web server to serve the files in `report_dir`
directory over HTTP (with TLS, by preference). They are static files,
which are easy to serve. Any web server can do it. You can copy the
files to another server too, if you prefer.

To test this, tell your CI node to seed the Radicle CI example
project. (You can also seed any other repository, but the example is
there to make it easy to try this out.) You will need to do this from
a different shell session (terminal window or tab) than the previous
command:

~~~sh
rad seed rad:z28U8KUBvVSMQc13NydX3LBDsdEdQ
~~~

Then push a patch to this repository, from your usual node, not the CI
node.

~~~sh
rad clone rad:z28U8KUBvVSMQc13NydX3LBDsdEdQ
cd radicle-ci-example
git switch -c patch
date > date
git add date
git commit -m trigger
git push rad HEAD:refs/patches
~~~

Check the report file to see that it works.

* `/home/_rad/native-ci.log`
* `/srv/http/index.html`
* With your web browser the URL to where the report directory gets
  published.

If all works, excellent. If not, and you need help, drop by the
[Radicle Zulip](https://radicle.zulipchat.com/) chat to ask for help.

Next, you probably want to consider what adapter you want to use. See
the
[`radicle-ci-integrations-docs`](https://explorer.radicle.gr/nodes/seed.radicle.gr/rad:z4Uh671FzoooaHjLvmtW9BtGMF9qm)
repository for a list.


## Installing Radicle CI with Ambient on Debian, automated

See the
[`install-ci.sh`](https://radicle.network/nodes/radicle.liw.fi/rad:zwTxygwuz5LDGBq255RA2CbNGrz8/tree/install-ci.sh) script in the `radicle-ci-broker` source
tree for a shell script to set up a Debian host as a CI node.

See the
[`ansible/install-ci.yml`](https://radicle.network/nodes/radicle.liw.fi/rad:zwTxygwuz5LDGBq255RA2CbNGrz8/tree/ansible/install-ci.yml) Ansible playbook and
accompanying role for the same thing, but using the Ansible configuration
management tool.

## Installing Radicle CI with Ambient on Debian, manually

* Make sure the following are installed:
  - `curl`
  - `xz` (in Debian, `xz-utils` package)
  - `genisoimage`
  - `xorriso`
* Install Radicle:
  - `curl -sSf https://radicle.dev/install | sh`
  - `rad auth`
  - `rad node start`
* Make sure the `kvm-intel` or `kvm-amd` kernel module is loaded, so
  that VM hardware acceleration works. You may need to add your user
  to the `kvm` group.
* Make sure the node has at least a test repository:
  - `rad seed rad:z28U8KUBvVSMQc13NydX3LBDsdEdQ`
* Install relevant software from `deb` packages on <https://radicle.dev/apt>:
  - download and install <https://files.radicle.dev/apt/radicle-archive-keyring.deb>
  - create `/etc/apt/sources.list.d/radicle_xyz_apt.list` with  
    `deb [signed-by=/usr/share/radicle/radicle-archive-keyring.asc] https://radicle.dev/apt release main`
* Download a VM image:
  - `curl https://files.liw.fi/ambient/ambient.qcow2.xz | unxz > ambient.qcow2`
* Create `~/ci-broker.yaml` (adjust paths so they refer to your home
  directory; make sure all the directories exist):

~~~yaml
db: /home/_rad/ci-broker.db
report_dir: /srv/http
queue_len_interval: 1min
adapters:
  ambient:
    command: /bin/radicle-ci-ambient
    env:
      RADICLE_CI_AMBIENT: /home/_rad/radicle-ci-ambient.yaml
triggers:
  - adapter: ambient
    filters:
    - !And
      - !HasFile ".radicle/ambient.yaml"
      - !Or
        - !DefaultBranch
        - !PatchCreated
        - !PatchUpdated
~~~

* Create `~/.config/ambient/config.yaml` (adjust the various fields
  for how much resource usage to allow, and paths):

~~~yaml
tmpdir: /tmp
projects: ~/ambient-projects.yaml
executor: /usr/bin/ambient-execute-plan
artifacts_max_size: 10G
cache_max_size: 50G
state: /home/_rad/ambient-state
qemu:
  cpus: 8
  memory: 16G
~~~

* Create `~/radicle-ci-ambient.yaml` (adjust paths again):

~~~yaml
image: /home/_rad/ambient.qcow2
logdir: /srv/http/ambient-log
log: /home/_rad/radicle-ci-ambient.log
base_url: "https://ci0.liw.fi//ambient-log"
~~~

* Create `/srv/http` with write permission for your user:
  `sudo install -d /srv/http -o $USER -g $USER`

* Start the CI broker in the foreground:
  - `cib --config ci-broker.yaml --log-level info process-events`
* In another terminal or shell sessions, trigger a run:
  - `cibtool --db ci-broker.db trigger --repo radicle-ci-example`
  - there should be several lines of log messages from `cib`
* Check result:
  - `cibtool --db ci-broker.db run list --json`


# Configuration

The CI broker must be started with a configuration file. The `config`
sub-command tells `cib` to write out the actual run-time
configuration, as computed from the specified configuration file and
built-in defaults.

~~~sh
cib --config /etc/radicle-ci-broker/config.yaml config
~~~

The output will be in JSON (which also works as YAML input):

~~~json
{
  "default_adapter": "dummy",
  "adapters": {
    "dummy": {
      "command": "../dummy.sh",
      "env": {},
      "sensitive_env": {}
    }
  },
  "filters": [],
  "triggers": null,
  "report_dir": "html",
  "db": "x.db",
  "max_run_time": {
    "secs": 3600,
    "nanos": 0
  },
  "queue_len_interval": {
    "secs": 3600,
    "nanos": 0
  }
}
~~~

The configuration fields are:

| field                            | summary                                                |
|:---------------------------------|:-------------------------------------------------------|
| `adapters`                       | list of CI adapters                                    |
| `concurrent_adapters`            | max number of adapters to run at the same time         |
| `db`                             | database                                               |
| `default_adapter`                | this will become deprecated, use `triggers` instead    |
| `filters`                        | this will become deprecated, use `triggers` instead    |
| `max_run_time`                   | maximum duration of a CI run                           |
| `queue_len_interval`             | how often the event queue length should be logged      |
| `report_dir`                     | directory where HTML and JSON report pages are written |
| `status_update_interval_seconds` | deprecated and ignored                                 |
| `triggers`                       | see [the "Triggering CI" chapter](#triggers)           |
| `update_job_cobs`                | create or update job COBs for CI runs?                 |

The items in the `adapters` list may use the following fields:

| field           | summary                                                                            |
|:----------------|:-----------------------------------------------------------------------------------|
| `command`       | name of command to run                                                             |
| `env`           | optional; a key/value map of environment variables to add when running the adapter |
| `sensitive_env` | optional; like `env`, but value are never logged                                   |
| `config`        | optional; a key/value map to configure the adapter                                 |
| `config_env`    | optional; name of environment variable for adapter configuration from `config`     |

The `config` and `config_env` fields allow embedding configuration
values for the adapter. The CI broker will write the values in
`config` to a temporary file, as YAML, and set the environment
variable named in `config_env` to the path to the temporary file. This
can reduce the number of files needed to be maintained to configure
`cib` and all the adapters.

## On concurrency

The CI broker allows mulitple CI runs to run at the same time. However, it
only allows at a time per repository. The `concurrent_adapters` setting refers
to the total number of adapter processes running at the same time.


# CI events

The CI broker currently supports a small set of CI events. There will
be more.

In the tables below, the fields have the following meanings:

* `from_node` -- the node from which the event originated
* `repo` -- the ID of the repository concerned
* `branch` -- the name of the branch created of updated
* `tip` -- the newest commit in the branch or patch
* `old_tip` -- the previous newest tip, before the change


## `BranchCreated`

A branch has been created. This may mean the repository has also been
created, but that is not certain.

| Event           | fields              | field types |
|:----------------|:--------------------|:------------|
| `BranchCreated` | `from_node`         | `NodeId`    |
|                 | `repo`              | `RepoId`    |
|                 | `branch`            | `BranchName`|
|                 | `tip`               | `Oid`       |

## `BranchUpdated`

A branch has been updated.

| Event           | fields      | field types |
|:----------------|:------------|:------------|
| `BranchUpdated` | `from_node` | `NodeId`    |
|                 | `repo`      | `RepoId`    |
|                 | `branch`    | `BranchName`|
|                 | `tip`       | `Oid`       |
|                 | `old_tip`   | `Oid`       |

## `BranchDeleted`

A branch has been deleted.

| Event           | fields      | field types |
|:----------------|:------------|:------------|
| `BranchDeleted` | `from_node` | `NodeId`    |
|                 | `repo`      | `RepoId`    |
|                 | `branch`    | `BranchName`|
|                 | `tip`       | `Oid`       |

## `TagCreated`

An annotated Git tag has been created.

| Event           | fields              | field types |
|:----------------|:--------------------|:------------|
| `TagCreated`    | `from_node`         | `NodeId`    |
|                 | `repo`              | `RepoId`    |
|                 | `tag_name`          | `RefString` |
|                 | `tip`               | `Oid`       |

## `TagUpdated`

An annotated Git tag has been updated.

| Event           | fields      | field types |
|:----------------|:------------|:------------|
| `TagUpdated`    | `from_node` | `NodeId`    |
|                 | `repo`      | `RepoId`    |
|                 | `tag_name`  | `RefString` |
|                 | `tip`       | `Oid`       |
|                 | `old_tip`   | `Oid`       |

## `TagDeleted`

An annotated Git tag has been deleted.

| Event           | fields      | field types |
|:----------------|:------------|:------------|
| `TagDeleted`    | `from_node` | `NodeId`    |
|                 | `repo`      | `RepoId`    |
|                 | `tag_name`  | `RefString` |
|                 | `tip`       | `Oid`       |

## `PatchCreated`

A patch has been created.

| Event          | fields      | field types |
|:---------------|:------------|:------------|
| `PatchCreated` | `from_node` | `NodeId`    |
|                | `repo`      | `RepoId`    |
|                | `patch`     | `PatchId`   |
|                | `new_tip`   | `Oid`       |

## `PatchUpdated`

A patch has been updated.

| Event          | fields      | field types |
|:---------------|:------------|:------------|
| `PatchUpdated` | `from_node` | `NodeId`    |
|                | `repo`      | `RepoId`    |
|                | `patch`     | `PatchId`   |
|                | `new_tip`   | `Oid`       |
|                |             |             |

# Event filters

The CI broker configuration can use the following conditions, and
AND/OR/NOT operators to build a filter expression: if the expression
evaluates as "true", the event is allowed and will trigger a CI run.
Otherwise it is discarded and does not trigger a CI run.

| Condition         | Meaning                                                   |
|:------------------|:----------------------------------------------------------|
| `Allow`           | Change is allowed                                         |
| `And` or `AllOf`  | Change is allowed if all the operands are true            |
| `AnyDelegate`     | Did change originate on a delegate node?                  |
| `BranchCreated`   | Branch was created                                        |
| `BranchDeleted`   | Branch was deleted                                        |
| `BranchUpdated`   | Branch was updated                                        |
| `TagCreated`      | Annotated tag was created                                 |
| `TagDeleted`      | Annotated tag was deleted                                 |
| `TagUpdated`      | Annotated tag was updated                                 |
| `Branch`          | Event refers to a specific Git branch                     |
| `DefaultBranch`   | Event refers to a default branch of the repository        |
| `Deny`            | Changes is not allowed                                    |
| `HasFile`         | Commit in event contains named file                       |
| `Node`            | Event originated from a specific node, identified by ID   |
| `Not` or `NoneOf` | Change is allowed is the operand expressions are is false |
| `Or` or `AnyOf`   | Change is allows if any of the operands is true           |
| `PatchCreated`    | Patch was created                                         |
| `PatchUpdated`    | Patch was updated                                         |
| `Patch`           | Event refers to a specific patch, identified by ID        |
| `Repository`      | Event refers to a specific repository, identified by ID   |

## Simple example

The following example is a snippet of YAML for the CI broker
configuration file to match events that refer to the` main` in the CI
broker repository.

~~~yaml
filters:
  - !And
    - !Repository "rad:zwTxygwuz5LDGBq255RA2CbNGrz8"
    - !Branch "main"
~~~

The conditions are expressed using the `!Foo` syntax in YAML. `Foo`
must be one of the operands from the table above. Simple values are
expressed as doubly quoted strings, and lists of operands are
sub-lists in YAML syntax.

The `filters` field is a list of filter expressions that are implicitly
joined together using '!Or` -- in other words, if any of the
expressions in the list allows the event, the whole list allows the
events.

## Complicated example

The following is the actual `cib` configuration on `ci0`, one of the author's
production CI nodes. It shows how to embed adapter configuration in the `cib`
configuration. Further, the Ambient CI configuration is extended in that.

The adapter configuratio sets `PATH` so that it includes
`/home/ci/.cargo/bin`, because the way that server is set up it doesn't
inherit that in its environment, when `cib` is run as a service via `systemd`.

~~~yaml
description: |
  <p>This is a Radicle CI node at <b>ci0</b>.
max_run_time: "60min"
db: /home/ci/ci.db
report_dir: /srv/http
concurrent_adapters: 5
update_job_cobs: false
adapters:
  ambient:
    command: /bin/radicle-ci-ambient
    env:
      PATH: /home/ci/.cargo/bin:/bin:/usr/local/bin
      RADICLE_CI_AMBIENT: /home/ci/radicle-ci-ambient.yaml
    config_env: RADICLE_CI_AMBIENT
    config:
      image: /home/ci/ambient.qcow2
      logdir: /srv/http/ambient-log
      log: /home/ci/adapter-logs
      base_url: "http://ci0/ambient-log"
      trust_repository_name: true
      allow_post_plan: true
      extra_ambient_config_values:
        rsync_target_base: "ci@http.liw.fi:/srv/http"
        rsync_target_map:
          ambient-ci: doc.liw.fi/ambient-ci
          cachedir: doc.liw.fi/cachedir
          clab: doc.liw.fi/clab
          ewww: doc.liw.fi/ewww
          jt: doc.liw.fi/jt
          obnam: doc.obnam.org/obnam
          obnam.org: obnam.org
          puomi: doc.liw.fi/puomi
          rad-ci: rad-ci.liw.fi
          radicle-ci-ambient: radicle-ci-ambient.liw.fi
          radicle-ci-broker: radicle-ci-broker.liw.fi 
          radicle-ci-homepage: radicle-ci.liw.fi
          riki: doc.liw.fi/riki
          sopass: doc.liw.fi/sopass
          sshca: doc.liw.fi/sshca
          subplot: doc.subplot.tech
          subplot-web: subplot.tech
          summain: doc.liw.fi/summain
          v-i: doc.liw.fi/v-i
          vmadm: doc.liw.fi/vmadm
          vmdb2: vmdb2-manual.liw.fi
        dput_target: apt.liw.fi
        secrets:
          - source: /home/ci/.ssh/id_rsa
            dest: ssh_key
          - source: /home/ci/.ssh/id_rsa-cert.pub
            dest: ssh_key-cert.pub
          - source: /etc/dput.cf
            dest: dput.cf
triggers:
  - adapter: ambient
    filters:
      - !And
        - !HasFile ".radicle/ambient.yaml"
        - !DefaultBranch
~~~

# Triggering CI {#triggers}

The Radicle CI broker can be configured to trigger CI runs in two
ways:

* a global filter and a default adapter
* a list of triggers, each of which has its own filter, and specifies
  an adapter to use

Example:

~~~yaml
adapters:
  foo:
    command: foo-adapter
    env:
      RADICLE_CI_FOO: foo-ci.yaml
  bar:
    command: bar-adapter
    env:
      RADICLE_CI_BAR: bar-ci.yaml
default_adapter: foo
filters:
  - !Branch "main"
triggers:
  - adapter: foo
    filters:
      - !Branch "staging"
  - adapter: bar
    filters:
      - !PatchCreated
      - !PatchUpdated
~~~

The above configuration specifies two aapters, `foo` and `bar`.

* `foo` is used for the `main` branch and `staging` branch
* `bar` is used for patches

Thus, if the `main` branch changes, the `foo` adapter is run. If a
branch other than `main` or `staging` changes, CI is not run.

## Configuring adapters

The `adapters` field in the configuration file gives a name to each
adapter. The name only matters within the configuration file, it has
no other significance, but each name must be different. For each
adapter the following fields are specified:

* `command` - the command by which the adapter is invoked
  - this is only the name of the command, without any options
  - the executable is found via the `PATH`, or the name can be an
    absolute path
* `env` is a mapping of name/value pairs; when the command is
  invoked, it is given each name as an environment variable, set to
  the value in the mapping
* `sensitive_env` is like `env`, but the values are never logged or
  output; this is to prevent accidental leaking of secrets

Both `env` and `sensitive_env` are optional.


## Default adapter

The `default_adapter` field can be set to the name of an adapter
in the `adapters` field. That adapter will be used if the `filters`
filters allow an event.

If `filters` isn't set, `default_adapter` is not used.


## Triggers

The `triggers` field contains a sequence conditions (filters) that are
used to decide if an adapter is to be run. A trigger has two fields:

* `filters` is a list of event filters, just like the top level
  `filters` field, with the same syntax, predicates, and meaning
* `adapter` names the adapter to run if the filters allow an event

If there are several triggers that are allowed, every adapter is run,
in sequence. If an earlier adapter run fails, the later ones are still
run. However, only the first error is returned to the CI broker.