Lars Wirzenius e82503bf95 fix(ci-broker.md): mock node control socket handling in helper
The test suite didn't work on Macs. A fundamental problem was using
"sleep 0.1" to wait for a background process to do something: this is
never a good idea. Either the wait time is too short, resulting in
spurious test failures, or it's so long it makes test take
unnecessarily long. I should have known better. A better approach is
to loop for a sufficiently long time and check if the background
process has done its thing every so often, and stop looping when it
has.

There were some other, more incidental incompatibility problems too.

To fix all this, replace use of "nc", a common tool with many
incompatible variants, by adding a "client mode" to
synthetic-events.rs that connects to the daemon and closes the
connection neatly, then waits for the daemon to delete the socket.
Also, change how it does what it does so that it works both Linux and
Mac. This includes removing setting a read timeout that failed on Mac,
but doesn't seem to be necessary now.

Also, change synthetic-events to log more and more detailed messages
to aid remote debugging of test suite failures in the future. The
daemon logs errors from fallible operations, instead of discarding
them and terminating, and the client outputs the daemon log if the
client fails, while waiting for the daemon to remove the socket.

Finally, during the furious debugging, refactor the code into smaller
functions so its easier to understand when nothing makes sense.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-29 12:58:10 +03:00
2024-08-26 09:31:57 +03:00
2024-05-14 16:34:20 +03:00
2024-03-31 11:16:58 +03:00
2024-08-26 09:31:57 +03:00

radicle-ci-broker

Add integration to CI engines/systems/services to Radicle, a distributed git hosting and collaboration system.

This is not quite production ready code yet, but it will eventually become a thing that listens for changes in a Radicle node, and triggers CI on the relevant ones.

Architecture

See the doc directory for architecture documentation. Quick summary: the CI broker gets events from the Radicle node, filters them based on its own config, and for any event that gets past the filter, runs the configured adapter executable. The broker and adapter use a simple JSON based message protocol over stdin/stdout. Each CI system has its own adapter.

To build the documentation, run make in the doc directory. You'll need some tools installed: Pandoc, graphviz (dot), PlantUML, pikchr-cli. The others are widely packaged, pikchr-cli is a Rust crate, so you can install it with cargo install pikchr-cli

Build and publish the documentation like this:

RADICLE_CI_BROKER_WEBROOT=/tmp/ci make -C doc publish

Binaries

The crate contains several binaries:

  • cib --- the actual CI broker
    • this is the only one you need to care about, the rest are for debugging
  • cibtool --- management tool for node operators, for managing the CI broker
  • synthetic-events --- test tool for CI broker developers

Packaging

There is simple, simplistic, rudimentary, personal-use-only packaging for Debian in the debian directory, used by Lars to build packages for his own use.

Running from the source tree

To run cib or cibtool from the source tree:

cargo run --bin cib -- --config config.yaml process-events
cargo run --bin cibtool -- --db ci-broker.db event list

Note the -- argument. It tells cargo run that all the arguments that follow are to be passed to the program being run, and not used by cargo itself.

Running tests

To run the test suite for the CI broker:

cargo test

(The usual way, for a Rust program.)

Configuration

The configuration file is named on the command line. It is a YAML file, for example:

default_adapter: native
db: ci-broker.sqlite
adapters:
  native:
    command: radicle-native-ci
    env:
      RADICLE_NATIVE_CI: /home/liw/radicle/radicle-native-ci/x/config.yaml
    sensitive_env:
      some_secret: some_secret_value_that_is_not_logged
filters:
  - !And
    - !Repository "rad:z2e6URdt1we1iG1BCVqtx8QVgsX4a"
    - !Or
      - !Branch "main"
      - !AnyPatchRef

Or if you only want to filter for patch COB updates (e.g. comments), to the specified repository:

default_adapter: native
db: ci-broker.sqlite
adapters:
  native:
    command: radicle-native-ci
    env:
      RADICLE_NATIVE_CI: /home/liw/radicle/radicle-native-ci/x/config.yaml
    sensitive_env:
      some_secret: some_secret_value_that_is_not_logged
filters:
  - !And
    - !Repository "rad:z2e6URdt1we1iG1BCVqtx8QVgsX4a"
    - !And
      - !AnyPatch
      - !Not 
        - !AnyPatchRef 

This runs the native CI engine as an adapter, on any repository events that pass the filter. The filter allows any changes to the main branch or any Radicle patch, on the specified repository.

License

Radicle CI broker is distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE and LICENSE-MIT for details.

S
Description
Radicle CI broker
Readme
5.6 MiB
Languages
Rust 96.8%
Nix 1.2%
Shell 1.2%
Makefile 0.3%
CSS 0.3%
Other 0.2%