Commit Graph
361 Commits
Author SHA1 Message Date
Lars Wirzenius b0dc2b9f83 fix: log message about thread ending shouldn't say it's starting
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-31 16:27:23 +03:00
Lars Wirzenius e82503bf95 fix(ci-broker.md): mock node control socket handling in helper
The test suite didn't work on Macs. A fundamental problem was using
"sleep 0.1" to wait for a background process to do something: this is
never a good idea. Either the wait time is too short, resulting in
spurious test failures, or it's so long it makes test take
unnecessarily long. I should have known better. A better approach is
to loop for a sufficiently long time and check if the background
process has done its thing every so often, and stop looping when it
has.

There were some other, more incidental incompatibility problems too.

To fix all this, replace use of "nc", a common tool with many
incompatible variants, by adding a "client mode" to
synthetic-events.rs that connects to the daemon and closes the
connection neatly, then waits for the daemon to delete the socket.
Also, change how it does what it does so that it works both Linux and
Mac. This includes removing setting a read timeout that failed on Mac,
but doesn't seem to be necessary now.

Also, change synthetic-events to log more and more detailed messages
to aid remote debugging of test suite failures in the future. The
daemon logs errors from fallible operations, instead of discarding
them and terminating, and the client outputs the daemon log if the
client fails, while waiting for the daemon to remove the socket.

Finally, during the furious debugging, refactor the code into smaller
functions so its easier to understand when nothing makes sense.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-29 12:58:10 +03:00
Lars Wirzenius 58ceb33620 fix: typo in log message
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-28 09:31:25 +03:00
Lars Wirzenius 609e33c13b feat: log adapter stderr output
The CI broker now logs the standard error output of the adapter, and
the adapter exit status, to the broker's own log. The log messages
look like this:

  {"msg":"adapter exit code","level":"DEBG","ts":"2024-08-27T17:38:26.762222386Z","exit_code":1}
  {"msg":"adapter stderr","level":"DEBG","ts":"2024-08-27T17:38:26.762289242Z","stderr":"woe be me\n"}

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-28 09:31:25 +03:00
Lars Wirzenius dd043e0154 feat: add --json to cibtool event list
Add a --json option to allow "cibtool event list" to output JSON, for
easier post-processing by other programs. Keep the --verbose, to avoid
making a breaking change, but hide it from --help output.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-27 19:14:50 +03:00
Lars Wirzenius 2c70b462bc tests(ci-broker.md): verify that cibtool checks repo exists
When adding an event, via "cibtool event add" or "cibtool trigger",
verify the tool checks that the repository the event is for actually
exists. The check was already there, this just adds a verification
scenario for that.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-27 18:25:15 +03:00
Lars Wirzenius 40cb793a54 fix: make test suite run on Macs
Change the radenv.sh script to use tilde expansion instead of getent.
Macs do not have a getent - that's a tool that's specific to glibc, so
it's mostly only available on Linux systems. But we don't need that:
Bash can do the expansion directly.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-26 14:39:24 +03:00
Lars Wirzenius c7cbcb0071 chore: update dependency on Subplot to 0.11.0
Also change required Rust version to be 1.80. This matches heartwood,
but is also necessary for Subplot dependencies.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-26 10:45:36 +03:00
Lars Wirzenius 968207a5f1 chore: prepare release 0.5.0
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-26 09:31:57 +03:00
Lars Wirzenius 0e4aeb832e docs(NEWS.md): fix heading level for previous release notes
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-26 09:10:16 +03:00
Lars Wirzenius 14aa655341 fix: heading markup typo
I missed this before merging. I was not careful.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-26 09:02:09 +03:00
Lars Wirzenius 2501340228 docs(ci-broker.md): update stakeholder groups: node-ops, devs,
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-23 16:16:53 +03:00
Lars Wirzenius 1da1308655 feat: add length of event queue to status.json
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-23 10:15:27 +03:00
Lars Wirzenius 85414e8673 feat: switch to structured logging only
Replace all use of log and env-logger crates with structured logging
using slog and slog-json.

Change the recently introduced logger module to use the slog-scope
crate. This avoids having to pass a reference to logger::Logging to
every place that needs to do any logging.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-23 10:03:49 +03:00
Lars Wirzenius f0a995900e fix: allow alias for Run::repo_name to mach old field name
This should allow painless upgrading from previous release.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-21 14:13:23 +03:00
Lars Wirzenius 07b3a48a5e feat: add table with current event queue to front report page
Add a table of pending events in the event queue. except for ones for
private repositories.

The public queue allows people to have a idea of how long they will
have to wait for CI to run for their change.

The table leaks the event queue, commit IDs, branch names, and patch
IDs, but if that's a problem, we can make this a table configurable,
on a per node basis.
2024-08-21 13:57:46 +03:00
Lars Wirzenius 2a40a12a9d Merge branch 'logging' 2024-08-20 09:56:12 +03:00
Lars Wirzenius 1e9f6c146d fix(ci-broker.md): fix scenarios to use new cibtool CLI
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-19 19:03:41 +03:00
Lars Wirzenius 5676cf8607 feat(src/bin/cibtoolcmd/run.rs)! simplify run add options
Change --repo to allow repository to be looked up by name, and
--commit to resolve a symbolic name to a full SHA, using the new util
module.

Drop the --alias option, it as misnamed. It was actually the
repository name, and we can look that up and don't need to have the
user provide it. Change the JSON output to say "repo_name" instead of
"repo_alias", for correctness.

This is a breaking change, because the command line interface and JSON
output change in an incompatible way.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-19 19:02:54 +03:00
Lars Wirzenius 05ab47af74 refactor(src/bin/cibtoolcmd/trigger.rs): use new util module
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-19 18:57:38 +03:00
Lars Wirzenius 0c6656f701 refactor: add a utility module
This module has code that various cibtool commands can share, for
example to look up a repository by its RID or name, if name is unique.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-19 18:56:04 +03:00
Lars Wirzenius 81a9a7b3b0 chore: prepare release 0.4.0
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-19 12:45:14 +03:00
Lars Wirzenius 13ce97b1f0 fix: "cibtool run add" sets the state if successful
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 17:26:49 +03:00
Lars Wirzenius e58d4c3af6 feat: use tables for repo list, run list
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 17:01:00 +03:00
Lars Wirzenius c808b03cab refactor: add a helper function to add a P with static text
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 11:59:34 +03:00
Lars Wirzenius 404a8dcd97 refactor: add a helper function to add an H1 element
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 11:57:59 +03:00
Lars Wirzenius 39b5c0a2fe refactor: create title only once
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 11:46:48 +03:00
Lars Wirzenius a34df9980f refactor: add helper to add the usual stuff to the HEAD element
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 11:45:15 +03:00
Lars Wirzenius 54029ae3e7 feat: move status.json report front page to the end
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-16 11:36:44 +03:00
Lars Wirzenius 7a11bedc00 feat: make sure the HTML and JSON files get created
This is the most rudimentary requirement. It does not verify they get
updated as the CI broker continues running and doing things, as that's
a little difficult to arrange a test for.

This revealed a problem in how the `cib process-events` threads work
together, so fix that too. Changed the page updater thread to output
the HTML report and JSON status file at least once, even if there were
not events to process. In addition, the event processing thread uses a
timeout to receive notification of new events, so that it doesn't get
stuck, if there aren't any events coming from the node control socket.

Added some logging to make it easier to debug these things in the
future.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-15 19:06:21 +03:00
Lars Wirzenius 413fd68642 refactor(cibtool): structure for subcommand implementation
Previously all subcommands for cibtool where in one file,
src/bin/cibtool.rs. That grew rather unwieldy and tiresome to
navigate. Split that into a module cibtoolcmd, with one sub-module per
top level subcommand for the leaf subcommands: the ones that do no
have subcommands themselves. Keep the rest of the subcommand structure
in cibtool.rs, so that it's easier to manage the subcommand structure
in one place.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-15 09:29:23 +03:00
Lars Wirzenius 402d58eea0 feat: start adding structured logging to cib
This is the smallest useful change to add structured logging. It adds
little useful logging, but builds a foundation for adding more useful
logging later, and for converging existing log messages to structured
one later.

Add some acceptance criteria for structured logging so that we do not
accidentally change the log messages we really, really care about.

Add dependencies on the slog and slog-json crates, which exist
precisely for structured logging.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-14 18:48:26 +03:00
Lars Wirzenius f75a78e82f fix(ci-broker.md): some stakeholder references
I failed to search and replace all instances of "Who: Lars". Fix that.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-14 18:22:02 +03:00
Lars Wirzenius 00b0d2bec0 docs: document stakeholders for the CI broker
Add stakeholder groups and change all scenarios to use those.

This is an initial set of stakeholders, to be built upon when the CI
broker starts maturing.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-14 14:25:29 +03:00
Lars Wirzenius f07da18fbc feat: add "run show" and "run update"
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-09 11:51:07 +03:00
Lars Wirzenius 2cc1dc4bca feat! "cibtool run add" run state option change
Make the "cibtool run add" options to set the state of a CI run be
more logical. There are now four:

* --triggered
* --running
* --success
* --failure

That is, drop the --running option. This allows the clap configuration
to specify that one of those four is required. I could not find a way
to specify that if --running was present, one of --success and
--failure was required, but that's OK: the --running option didn't
actually help to make things easier for the user.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-09 10:40:52 +03:00
Lars Wirzenius a80ab49d7d chore: bump dependency on html-page crate 2024-08-08 17:53:40 +03:00
Lars Wirzenius 8ee42c7cc2 chore: switch from serde-yaml to serde-yml
serde-yml is maintainted, serde-yaml isn't.
2024-08-08 17:52:16 +03:00
Lars Wirzenius bdad79d442 feat: have threads notify each other of needing to do something
Event adding process notifies event processing thread that there are
new events, and the event processing thread notifies the report page
generation thread that a new run has finished. In addition, the
reporting thread reports after a while whether there are any new runs
or not, so that the page time stamp gets updated.

Also drop some now-unused code.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:35:14 +03:00
Lars Wirzenius 2414e342d1 feat: add NotificationChannel
This will be useful for threads to notify each other that they have
work to do. The notification does not carry any information, such as
an event or a run, that goes via the database, for robustness and
simplicity.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:34:15 +03:00
Lars Wirzenius d5159a7400 feat: whenever run info changes, update it in database
Especially, when adapter reports the run ID it assigns or other
information.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:33:47 +03:00
Lars Wirzenius 9a4a39f73b feat(src/pages.rs): tweak per-repo html report with both run ids
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:33:47 +03:00
Lars Wirzenius 2317a19fd0 feat(src.pages.rs: both run ids in repo list for latest run
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:33:47 +03:00
Lars Wirzenius eeb79ef090 feat(src/msg.rs): add RunId::as_str method
This makes it easier to use the run ID as a string, without creating a
whole new String value.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:33:27 +03:00
Lars Wirzenius 9d912e6ee6 feat(src/bin/cibtool.rs): list broker run id with "run list"
Show both the broker and adapter run IDs in the output of "cibtool run
list".

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:32:37 +03:00
Lars Wirzenius 6dadbcd27d feat(src/bin/cibtool.rs)! make --id option to "run add" optional
Previously we always chose a random adapter run id, now use what's
given, if any.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:32:37 +03:00
Lars Wirzenius a4a03b414a feat: add broker run id to Run
Add a unique, immutable broker run ID to each run, in addition to the
adapter run ID that may not always be available, and is not guaranteed
to be unique.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-08-02 18:31:36 +03:00
Lars Wirzenius 17828919ae fix(src/bin/cibtool.rs): ensure ref change event has a base commit
The trigger message given to the CI adapter to start a CI run must
have a list of commits. This requires the "ref changed" event to have
a base commit. The commits after the base commit to the tip commit in
the event become the list of commits in the trigger message.

"cibtool trigger" is given the tip commit with the "--commit" option.
Change it to use a parent commit of that as the base commit, or if
there is no parent commit (i.e., there is only the one commit in the
repository), the commit itself.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-07-31 09:55:19 +03:00
Lars Wirzenius 105a09a8db feat(src/msg.rs)! don't panic on missing commit or unknown type
Change Request::commit to not panic if the push or patch data has no
commits, or if the request is neither push nor patch. Instead, return
an appropriate error. A panic message isn't very helpful to users;
errors can have helpful messages.

This changes the method signature of an exported method and that is a
breaking change.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-07-31 09:54:56 +03:00
Lars Wirzenius a3fbb6e0e4 fix(debian/conrol): fix typo in package description
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-07-31 09:02:25 +03:00