Change the logical negation operator `Not` to take a list of filters
as operands. The operator returns true, if and only if all of the
operands are false. The syntax is now consistent with logical `And`
and `Or`, which simplifies code, and is expected to simplify use.
In YAML syntax, `Not` now works like this:
~~~
!Not
- !Allow
~~~
Previously, `!Not` only had one operand, so that above would have been
expressed as `!Not Allow`.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
"cibtool trigger" option "--ref" defaults to "main" and "--commit" to
"HEAD", and there's no need to say those explicitly.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
The default doesn't change, but it's now called
`DEFAULT_QUEUE_LEN_INTERVAL` instead of `DEFAULT_MAX_RUN_TIME`.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
For several releases already, `cib` has refused to start if the
database has any "broker events". Broker events were a predecessor for
CI events, and they haven't been supported by the CI broker since
0.7.0, from October 2024.
Drop the Db method to query the database for queued broker events.
Also drop the creation of a table for that queue.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
A trigger is a condition (filter) when to run CI using a specific
adapter. Change the configuration to allow them, and make the old list
of filters and default adapter be optional. This way, the user can use
either or both.
CI is run for each of:
- top level filter matches, using default adapter
- trigger filter matches, using the adapter specified in the trigger
To be clear, this can mean that CI is run many times for one CI event.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Construct the collection, and give it to the queue processor, which is
the only part that needs to know what adapters there are.
For now, only the default adapter is used, but this makes a space from
which to pick a different for each CI run.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This is preparation for allowing many triggers: we will want to allow
any number of triggers allow a single change, resulting in a different
CI run for each. Change the code that picks an adapter to be able to
pick any number of them, and then run all of them. Even if one fails,
run the rest.
This will potentially mean run the same adapter many times, although
it's meant to allow running a different adapter for each trigger.
Running the same adapter many times may be pointless, but we can't
reliably know that, only the user can. It depends on the adapter, the
change, and what the user is trying to achieve. So for now we don't
worry about that. If it turns out to be important, we can later add a
way to say each adapter will only be run once.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Move CI event filtering from the queue adder to the queue processor.
This is ground work for supporting different adapters for different
runs.
Later, each "trigger" will have a separate filter, and if the filter
allows an event, the adapter for that trigger is run. However, we have
no way to communicate the chosen adapter via the event queue in the
database, without making a breaking database schema, which I would
prefer to avoid.
Instead, we add all events into the queue, and filter when we remove
them from the queue. This may be a little wasteful in terms of disk
space and I/O, if there are very large numbers of events. If this
turns out to be too expensive, we'll revert this change and do a
breaking schema change.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Remove the default adapter from Broker, and make callers give the
adapter to use to the Broker::execute_ci method each time. This is
ground work for allowing different adapters for different CI runs.
There's no need for Broker to know about any adapters, except the one
it is currently running.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
The broker overall only supports the default adapter. When we add
support for using an adapter other than the default one, it'll not be
per repository.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Make struct Config fields private and introduces getter methods to
access them. This will make it easier to change the CI broker to be
able to run different adapters for different changes.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
The configuration file can have a field `queue_len_interval`, and
defaults to 10 seconds. This should reduce log spam.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Previously, we failed to capture all of stderr from the adapter if the
adapter sent messages to its stdout could not be parsed, or otherwise
meant the CI broker deduced the CI run had failed. Now we read all of
stderr (module time limits) before returning an error result.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Add subcommand "cibtool log" to extract cib log messages from journald
logs, and pretty print them. Optionally with some very rudimentary
filtering based on log level, message kind, or broker run id.
Example:
sudo journalctl -S today -u radicle-ci-broker -o json | cibtool log --log-level info
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This identifies what the message is, without having to parse the
message text, which is free form and may change and may some day get
translated.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Adjust log levels for messages, move data from free-form text field
into extra fields (even if still in free-form text form, as debug
formatted Rust values), and other related changes.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Change CI event creation to set the `from_node` field to the node id
in the Git ref being updated. Previously we set it to the `remote`
field in the node event, but that is id of the node from which our
local node got the `RefsUpdated` event, not the node where the change
was originally made.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This makes the code more verbose, possibly harder to follow, but will
make it easier to figure what's going wrong, in production. That's
more valuable than elegant code.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Refactor the code the constructs CI events from node events, and
trigger messages from CI events, to debug why I'm having trouble with
patch events.
Also adjust debug logging so I have more insight of what's going on.
Also, make code clearer, less repetitive, and overall easier to
follow.
Add support to "cibtool event add" for branch deletion, and patch
creation and update events. Use this in a scenario that adds every
type of event to the queue and processes them. This ensures the CI
broker can actually handle any kind of CI event.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
`subplot docgen` fails if an embedded file is not used. Drop the three
files that are no longer used.
This would've been found earlier if I gated merges on CI passing.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This will help with post-processing and filtering logs without having
to rely on the text of the log message.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Switch from the "slog" family of crates for structured logging to the
"tracing" family. There should be no significant changes to the log
messages, so I don't consider this to be a breaking change.
The benefit of this change is that it opens up the possibility to add
extra fields to the log messages as JSON values, instead of strings
that contain the values as JSON.
With slog we currently have this message::
{"msg":"broker database: /home/_rad/ci-broker.db"}
With slog or tracing we can change this to:
{"msg":"open database", "filename": "/home/_rad/ci-broker.db"}
With tracing we can have this:
{"msg":"open database","db":{"type":"sqlite","filename":"/home/_rad/ci-broker.db","size":12765}}
which would be, formatted prettily
{
"msg": "open database",
"db": {
"type": "sqlite",
"filename": "/home/_rad/ci-broker.db",
"size": 12765
}
}
Such extra fields can then be easily queried or otherwise processed
with programs, much more easily than having such programs parse text
fields as JSON before they can be processed.
Doing this with slog is supposed to be possible, but I have failed to
get it to work. The slog and tracing families of libraries seem
otherwise similar for our purposes.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
We used to parse the ref name to find out the branch name, when
creating a trigger message. The CI events now do that for us. This
meant that parsing the branch name failed, and resulted in the empty
string.
Fix this by using already-parsed branch name from the CI event instead
of trying to parse it again.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Lars Wirzenius