radicle-ci-broker/doc/userguide.md

Introduction

The Radicle CI broker runs CI for repositories in the local Radicle node. This is the user guide for the CI broker.

The CI broker helps users run validation on changes to their software project, by automating the building and testing of the projects when anything in the repository changes. This is often called "continuous integration".

(Technically, "continuous integration" is the software development practice to merge changes into the main line of development frequently, at least daily, to avoid painful merge conflict resolutions. However, for this guide we say "CI" to mean "when repository changes, perform these actions", which is a more generic, and quite popular definition, if not very purist.)

Overview

The Radicle node stores Git repositories and synchronizes them with other Radicle nodes. The CI broker connects to its local node and gets "node events" whenever anything changes in the node. The relevant change for the CI broker is that a Git references ("refs") in a repository have been created, updated, or deleted. For now, these are branches. Later, Radicle and the CI broker will support other references, such as tags.

There are no node events for Git repositories being created or deleted. It's not possible to create a Radicle repository without creating a branch, so just looking at references is enough.

The CI broker looks at the reference changes and refines them into "CI events", which are more suitable for the kind CI use that the CI broker is meant to enable, than "this ref changed", which is quite low level.

The CI events are filtered, and events that are allowed by the filter trigger a CI run, which runs another program called the CI adapter. The adapter arranges for a CI system or CI engine to execute CI for the code change captured in the CI event. There are adapters for different CI systems, such as GitHub Actions, Concourse, Kraken, and also the "native adapter", that runs a shell script locally on the host where the adapter runs. Each adapter may require a different configuration to suit the CI system it targets.

CI events

The CI broker currently supports a small set of CI events. There will be more.

In the tables below, the fields have the following meanings:

• from_node -- the node from which the event originated
• repo -- the ID of the repository concerned
• branch -- the name of the branch created of updated
• tip -- the newest commit in the branch or patch
• old_tip -- the previous newest tip, before the change

BranchCreated

A branch has been created. This may mean the repository has also been created, but that is not certain.

Event	fields	field types
BranchCreated	from_node	NodeId
	repo	RepoId
	branch	RefSting
	tip	Oid

BranchUpdated

A branch has been updated.

Event	fields	field types
BranchUpdated	from_node	NodeId
	repo	RepoId
	branch	RefSting
	tip	Oid
	old_tip	Oid

BranchDeleted

A branch has been deleted.

Event	fields	field types
BranchDeleted	from_node	NodeId
	repo	RepoId
	branch	RefString
	tip	Oid

PatchCreated

A patch has been created.

Event	fields	field types
PatchCreated	from_node	NodeId
	repo	RepoId
	patch	PatchId
	new_tip	Oid

PatchUpdated

A patch has been updated.

Event	fields	field types
PatchUpdated	from_node	NodeId
	repo	RepoId
	patch	PatchId
	new_tip	Oid

Event filters

The CI broker configuration can use the following conditions, and AND/OR/NOT operators to build a filter expression: if the expression evaluates as "true", the event is allowed and will trigger a CI run. Otherwise it is discarded and does not trigger a CI run.

Condition	Meaning
Allow	Change is allowed
And or AllOf	Change is allowed if all the operands are true
BranchCreated	Branch was created
BranchDeleted	Branch was deleted
BranchUpdated	Branch was updated
Branch	Event refers to a specific Git branch
DefaultBranch	Event refers to a default branch of the repository
Deny	Changes is not allowed
HasFile	Commit in event contains named file
Node	Event originated from a specific node, identified by ID
Not or NoneOf	Change is allowed is the operand expressions are is false
Or or AnyOf	Change is allows if any of the operands is true
PatchCreated	Patch was created
PatchUpdated	Patch was updated
Patch	Event refers to a specific patch, identified by ID
Repository	Event refers to a specific repository, identified by ID

Example

The following example is a snippet of YAML for the CI broker configuration file to match events that refer to the main in the CI broker repository.

filters:
  - !And
    - !Repository "rad:zwTxygwuz5LDGBq255RA2CbNGrz8"
    - !Branch "main"

The conditions are expressed using the !Foo syntax in YAML. Foo must be one of the operands from the table above. Simple values are expressed as doubly quoted strings, and lists of operands are sub-lists in YAML syntax.

The filters field is a list of filter expressions that implicitly joined together using '!Or` -- in other words, if any of the expressions in the list allows the event, the whole list allows the events.

Triggering CI

The Radicle CI broker can be configured to trigger CI runs in two ways:

• a global filter and a default adapter
• a list of triggers, each of which has its own filter, and specifies an adapter to use

Example:

adapters:
  foo:
    command: foo-adapter
    env:
      RADICLE_CI_FOO: foo-ci.yaml
  bar:
    command: bar-adapter
    env:
      RADICLE_CI_BAR: bar-ci.yaml
default_adapter: foo
filters:
  - !Branch "main"
triggers:
  - adapter: foo
    filters:
      - !Branch "staging"
  - adapter: bar
    filters:
      - !PatchCreated
      - !PatchUpdated

The above configuration specifies two aapters, foo and bar.

• foo is used for the main branch and staging branch
• bar is used for patches

Thus, if the main branch changes, the foo adapter is run. If a branch other than main or staging changes, CI is not run.

Configuring adapters

The adapters field in the configuration file gives a name to each adapter. The name only matters within the configuration file, it has no other significance, but each name must be different. For each adapter the following fields are specified:

• command - the command by which the adapter is invoked
  • this is only the name of the command, without any options
  • the executable is found via the PATH, or the name can be an absolute path
• env is a mapping of name/value pairs; when the command is invoked, it is given each name as an environment variable, set to the value in the mapping
• sensitive_env is like env, but the values are never logged or output; this is to prevent accidental leaking of secrets

Both env and sensitive_env are optional.

Default adapter

The default_adapter field can be set to the name of an adapter in the adapters field. That adapter will be used if the filters filters allow an event.

If filters isn't set, default_adapter is not used.

Triggers

The triggers field contains a sequence conditions (filters) that are used to decide if an adapter is to be run. A trigger has two fields:

• filters is a list of event filters, just like the top level filters field, with the same syntax, predicates, and meaning
• adapter names the adapter to run if the filters allow an event

If there are several triggers that are allowed, every adapter is run, in sequence. If an earlier adapter run fails, the later ones are still run. However, only the first error is returned to the CI broker.