The test suite didn't work on Macs. A fundamental problem was using
"sleep 0.1" to wait for a background process to do something: this is
never a good idea. Either the wait time is too short, resulting in
spurious test failures, or it's so long it makes test take
unnecessarily long. I should have known better. A better approach is
to loop for a sufficiently long time and check if the background
process has done its thing every so often, and stop looping when it
has.
There were some other, more incidental incompatibility problems too.
To fix all this, replace use of "nc", a common tool with many
incompatible variants, by adding a "client mode" to
synthetic-events.rs that connects to the daemon and closes the
connection neatly, then waits for the daemon to delete the socket.
Also, change how it does what it does so that it works both Linux and
Mac. This includes removing setting a read timeout that failed on Mac,
but doesn't seem to be necessary now.
Also, change synthetic-events to log more and more detailed messages
to aid remote debugging of test suite failures in the future. The
daemon logs errors from fallible operations, instead of discarding
them and terminating, and the client outputs the daemon log if the
client fails, while waiting for the daemon to remove the socket.
Finally, during the furious debugging, refactor the code into smaller
functions so its easier to understand when nothing makes sense.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
The CI broker now logs the standard error output of the adapter, and
the adapter exit status, to the broker's own log. The log messages
look like this:
{"msg":"adapter exit code","level":"DEBG","ts":"2024-08-27T17:38:26.762222386Z","exit_code":1}
{"msg":"adapter stderr","level":"DEBG","ts":"2024-08-27T17:38:26.762289242Z","stderr":"woe be me\n"}
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Add a --json option to allow "cibtool event list" to output JSON, for
easier post-processing by other programs. Keep the --verbose, to avoid
making a breaking change, but hide it from --help output.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
When adding an event, via "cibtool event add" or "cibtool trigger",
verify the tool checks that the repository the event is for actually
exists. The check was already there, this just adds a verification
scenario for that.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Change the radenv.sh script to use tilde expansion instead of getent.
Macs do not have a getent - that's a tool that's specific to glibc, so
it's mostly only available on Linux systems. But we don't need that:
Bash can do the expansion directly.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Also change required Rust version to be 1.80. This matches heartwood,
but is also necessary for Subplot dependencies.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Replace all use of log and env-logger crates with structured logging
using slog and slog-json.
Change the recently introduced logger module to use the slog-scope
crate. This avoids having to pass a reference to logger::Logging to
every place that needs to do any logging.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Add a table of pending events in the event queue. except for ones for
private repositories.
The public queue allows people to have a idea of how long they will
have to wait for CI to run for their change.
The table leaks the event queue, commit IDs, branch names, and patch
IDs, but if that's a problem, we can make this a table configurable,
on a per node basis.
Change --repo to allow repository to be looked up by name, and
--commit to resolve a symbolic name to a full SHA, using the new util
module.
Drop the --alias option, it as misnamed. It was actually the
repository name, and we can look that up and don't need to have the
user provide it. Change the JSON output to say "repo_name" instead of
"repo_alias", for correctness.
This is a breaking change, because the command line interface and JSON
output change in an incompatible way.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This module has code that various cibtool commands can share, for
example to look up a repository by its RID or name, if name is unique.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This is the most rudimentary requirement. It does not verify they get
updated as the CI broker continues running and doing things, as that's
a little difficult to arrange a test for.
This revealed a problem in how the `cib process-events` threads work
together, so fix that too. Changed the page updater thread to output
the HTML report and JSON status file at least once, even if there were
not events to process. In addition, the event processing thread uses a
timeout to receive notification of new events, so that it doesn't get
stuck, if there aren't any events coming from the node control socket.
Added some logging to make it easier to debug these things in the
future.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Previously all subcommands for cibtool where in one file,
src/bin/cibtool.rs. That grew rather unwieldy and tiresome to
navigate. Split that into a module cibtoolcmd, with one sub-module per
top level subcommand for the leaf subcommands: the ones that do no
have subcommands themselves. Keep the rest of the subcommand structure
in cibtool.rs, so that it's easier to manage the subcommand structure
in one place.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This is the smallest useful change to add structured logging. It adds
little useful logging, but builds a foundation for adding more useful
logging later, and for converging existing log messages to structured
one later.
Add some acceptance criteria for structured logging so that we do not
accidentally change the log messages we really, really care about.
Add dependencies on the slog and slog-json crates, which exist
precisely for structured logging.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Add stakeholder groups and change all scenarios to use those.
This is an initial set of stakeholders, to be built upon when the CI
broker starts maturing.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Make the "cibtool run add" options to set the state of a CI run be
more logical. There are now four:
* --triggered
* --running
* --success
* --failure
That is, drop the --running option. This allows the clap configuration
to specify that one of those four is required. I could not find a way
to specify that if --running was present, one of --success and
--failure was required, but that's OK: the --running option didn't
actually help to make things easier for the user.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Event adding process notifies event processing thread that there are
new events, and the event processing thread notifies the report page
generation thread that a new run has finished. In addition, the
reporting thread reports after a while whether there are any new runs
or not, so that the page time stamp gets updated.
Also drop some now-unused code.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This will be useful for threads to notify each other that they have
work to do. The notification does not carry any information, such as
an event or a run, that goes via the database, for robustness and
simplicity.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Add a unique, immutable broker run ID to each run, in addition to the
adapter run ID that may not always be available, and is not guaranteed
to be unique.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
The trigger message given to the CI adapter to start a CI run must
have a list of commits. This requires the "ref changed" event to have
a base commit. The commits after the base commit to the tip commit in
the event become the list of commits in the trigger message.
"cibtool trigger" is given the tip commit with the "--commit" option.
Change it to use a parent commit of that as the base commit, or if
there is no parent commit (i.e., there is only the one commit in the
repository), the commit itself.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Change Request::commit to not panic if the push or patch data has no
commits, or if the request is neither push nor patch. Instead, return
an appropriate error. A panic message isn't very helpful to users;
errors can have helpful messages.
This changes the method signature of an exported method and that is a
breaking change.
Signed-off-by: Lars Wirzenius <liw@liw.fi>