radicle-ci-broker/doc/architecture.md

# Overview

[Radicle]: https://radicle.xyz/
[Git]: https://git-scm.com/
[CI]: https://en.wikipedia.org/wiki/Continuous_integration

[Radicle] is a peer-to-peer collaboration system
built on top of the [Git] version control system. Radicle has support
for integrating with continuous integration ([CI]) systems, using an
architecture where a "broker" listens to events about changes to
repositories stored in a node, and launching the appropriate "adapter"
for each change, according to its configuration.

This means each node can opt into running CI for what projects and
changes according to the interests of the person whose node it is.

* The delegates for a repository might run CI on all patches to make
  merge decisions with more confidences.
* Someone else, who is contributing to a project, might only care about
  patches they themselves created, and only run CI for those.
* A third party might run CI for projects they use, to know if it's OK
  to deploy to their production systems.

Radicle provides its own, very simple "native CI" solution. It's just
good enough for the Radicle project to use itself. In addition, there
are adapters that integrate with external CI systems.

## Goal of Radicle CI

Context: The user is a software developer working on a project that
uses Radicle for version control. The project has an automated test
suite, and in-repository configuration for how to build the project
and run the test suite, in a format suitable for the CI engine being
used.

In the long run, the goal for CI in Radicle is "anything that makes it
easier, more fun, and faster to produce working software", but that's
not a concrete goal.

At this stage in the development of Radicle CI has two concrete goals:

* When I create a patch to propose a change, I am automatically told
  if the project branch with my committed changes fails to build or
  pass its test suite. I can also manually check what the status of
  that process ("CI run") is, and find out what the build log is, to
  investigate any problems.

  - This is "build and test the patch branch".

* When a project delegate merges my patch, both they and I are
  automatically told if the merge fails due to a merge conflict, or
  if, after the merge the project no longer builds or its test suite
  fails.

  - This is "build and test the master branch after the merge". This
    is useful, because sometimes a merged change breaks the build or
    the test suite, even when there are no merge conflicts.

It is not yet clear how notifications will work.

## Components of Radicle CI

Using CI with Radicle requires several co-operating components:

* The Radicle node (`radicle-node`).
* The Radicle CI broker (`cib`).
* An adapter the runs CI, possibly using an external CI service.
* A management tool (`cibtool`).

~~~pikchr
right
Node: ellipse "radicle-node" fill Beige
move
move
Cib: ellipse "cib" fill Azure
down
move
Adapter: box "adapter" fill Azure
move
External: box "external" "CI" "system" fill Aquamarine

move right from Cib.e then then right then right then up
Db: cylinder "SQLite" "db" fill Pink

move up Cib.height from Cib.n
Cibtool: ellipse "cibtool" fill Azure
arrow thin from Cibtool.e to Db.nw <->

move from Db.s down 150%
Logs: cylinder height 200% "CI run" "logs" "reports" fill Pink

arrow thin from Node.e to Cib.w "node" above "event" below
arrow thin from Cib.ne to Db.sw <->
arrow dotted from Cib.e to Logs.nw
arrow dotted from Adapter.e to Logs.sw

Req: arc thin dashed from Cib.sw to Adapter.nw ->
move left 15% from Req.c
text "stdin" small "trigger" small

Resp: arc thin dashed from Adapter.ne to Cib.se ->
move right 15% from Resp.c
text "stdout" small "result" small

arrow thin dashed from External.n to Adapter.s <->
~~~

The node emits an event (`RefsFetched`), when it has fetched updates
to one of the repositories it has. The event specifies the repository,
and every Git ref. The CI broker parses this event to extract
information about the kind of change: branch has created? updated?
deleted? Likewise for tags and patches, etc. These become "CI events",
to distinguish them from "node events".

The node operator can specify filters on CI events, and if an event
gets past a filter, the CI broker will run an external program called
an "adapter". The adapter has the responsibility of running CI for the
change, and to report to the CI broker if it succeeded or failed.

Communication with the adapter is via the adapter's standard input
and output, using single-line JSON messages.

~~~plantuml
@startuml
node -> cib : RefsFetched event
cib -> adapter : spawn
cib -> adapter : send request
adapter-> engine : trigger run
engine -> adapter : response with run id
adapter -> cib : response: run id
engine -> adapter : result of run
adapter -> cib : response: result
@enduml
~~~


The CI broker also generates HTML report pages to list all the CI runs
it has performed.

# The adapter

The adapter process reads the request to perform a CI run on a
specific commit in a repository, and responds with the id of the run,
then later with the status of the finished run.

For native CI, the adapter actually is the CI engine and performs the
CI run itself. For external CI, the adapter process does whatever it
needs to do to get the external CI engine instance to perform the CI
run. If the CI engine calls back via web hook to notify of the run
finishing, the adapter process needs to receive the call and process
it.

External CI engines allow complex pipelines to be written and support 
a variety of workflows. Different jobs or tasks can be triggered
based on different events (e.g. `push`, `patch created`, `patch 
updated`, etc.), to satisfy different workflow needs. (Only a few of
these are actually implemented yet, but the scaffolding to support
more is there.)

Some examples of real-world use-cases:

- trigger "fast" tests on every push to any branch
- trigger "relatively fast" tests only when a Patch is created / 
updated
- trigger "full test suite" on every push to the default branch (e.g. 
`main`)

In order to allow developers using Radicle the same flexibility that
they are used to on other forges, we want the broker to pass on 
whatever information it already has from the node events to the 
adapters, so they can pass it on to external CI systems, as appropriate. 

# Report generation

The CI broker has an SQLite database file for persistent storage of
information of the CI runs it triggers. This is used to generate
report pages, among other things.

The report page generation is done in its own thread, separate from
the main thread of the CI broker. This allows the reporting to happens
independently of what the main thread is doing. In particular, it
means the report generation happens even while the main thread is busy
running an adapter.

When it comes to per-run logs, the adapter can include a URL to one in
the first response message. The URL will be included as a link in the
report HTML.