Commit Graph
493 Commits
Author SHA1 Message Date
Lars Wirzenius 2e2236aea7 fix: use node id from ref name as originating node id
Change CI event creation to set the `from_node` field to the node id
in the Git ref being updated. Previously we set it to the `remote`
field in the node event, but that is id of the node from which our
local node got the `RefsUpdated` event, not the node where the change
was originally made.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-05 08:20:04 +02:00
Lars Wirzenius 10f52163ba feat: add detailed trace logging to trigger message construction
This makes the code more verbose, possibly harder to follow, but will
make it easier to figure what's going wrong, in production. That's
more valuable than elegant code.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-05 08:20:04 +02:00
Lars Wirzenius 660109e72e feat: use trace level for trigger creation logging
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-05 08:20:04 +02:00
Lars Wirzenius 3d942273f9 feat: extract config data to a separate field in log message
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-05 08:20:04 +02:00
Lars Wirzenius bb9e9f27bc refactor: node and CI event and trigger message creation
Refactor the code the constructs CI events from node events, and
trigger messages from CI events, to debug why I'm having trouble with
patch events.

Also adjust debug logging so I have more insight of what's going on.

Also, make code clearer, less repetitive, and overall easier to
follow.
2024-12-05 08:20:04 +02:00
Lars Wirzenius 1bc0ad40dc feat: cibtool can add every type of CI event to event queue
Add support to "cibtool event add" for branch deletion, and patch
creation and update events. Use this in a scenario that adds every
type of event to the queue and processes them. This ensures the CI
broker can actually handle any kind of CI event.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-05 08:20:04 +02:00
Lars Wirzenius 46651b7fa5 chore: bump dependency on radicle to 0.14.0
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-05 08:20:04 +02:00
Lars Wirzenius ac3fcf28a3 doc: add help texts to cib options and subcommands
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-04 11:01:49 +02:00
Lars Wirzenius f0a3e65c59 fix: drop unused embedded files from the subplot document
`subplot docgen` fails if an embedded file is not used. Drop the three
files that are no longer used.

This would've been found earlier if I gated merges on CI passing.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-12-04 10:59:47 +02:00
Lars Wirzenius 4512912804 test: document and test step to construct a refsFetched event
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-29 10:02:47 +02:00
Lars Wirzenius 4589438f46 test: document and test step to create a repository
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-29 10:02:47 +02:00
Lars Wirzenius d802be2a69 test: document and test the step to create and configure a node
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-29 10:02:47 +02:00
Lars Wirzenius d7428e5f94 chore: drop unused scenario step implementations
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-29 10:02:47 +02:00
Lars Wirzenius a1ff0a41b7 test: use new scenario steps in all scenarios
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-29 10:02:47 +02:00
Lars Wirzenius 964d3e1ed2 test: add higher level scenario steps
These are q step towards making scenarios less mired with test
implementation details.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-29 10:02:44 +02:00
Lars Wirzenius b4f379ad26 feat: use fields for extra data in log messages
This makes it easier to access the data than having to parse it from
free-form text fields.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-27 13:43:22 +02:00
Lars Wirzenius ee177c8f20 chore: drop unnecessary Logger
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-27 13:23:32 +02:00
Lars Wirzenius a506c6ebd3 chore: drop unused LogWriter
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-27 13:23:26 +02:00
Lars Wirzenius 9727a392aa feat: add a "kind" field to log messages
This will help with post-processing and filtering logs without having
to rely on the text of the log message.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-27 13:06:33 +02:00
Lars Wirzenius cf395350ea feat: use the tracing crate for structured logging
Switch from the "slog" family of crates for structured logging to the
"tracing" family. There should be no significant changes to the log
messages, so I don't consider this to be a breaking change.

The benefit of this change is that it opens up the possibility to add
extra fields to the log messages as JSON values, instead of strings
that contain the values as JSON.

With slog we currently have this message::

  {"msg":"broker database: /home/_rad/ci-broker.db"}

With slog or tracing we can change this to:

  {"msg":"open database", "filename": "/home/_rad/ci-broker.db"}

With tracing we can have this:

  {"msg":"open database","db":{"type":"sqlite","filename":"/home/_rad/ci-broker.db","size":12765}}

which would be, formatted prettily

  {
    "msg": "open database",
    "db": {
      "type": "sqlite",
      "filename": "/home/_rad/ci-broker.db",
      "size": 12765
    }
  }

Such extra fields can then be easily queried or otherwise processed
with programs, much more easily than having such programs parse text
fields as JSON before they can be processed.

Doing this with slog is supposed to be possible, but I have failed to
get it to work. The slog and tracing families of libraries seem
otherwise similar for our purposes.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-26 18:57:25 +02:00
Lars Wirzenius 84cfab01de feat: lower log level for various messages
Log level info should not be full of irrelevancies for node operators.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-25 10:18:20 +02:00
Lars Wirzenius 67c215c9d8 fix: show branch name in push events in HTML reports
We used to parse the ref name to find out the branch name, when
creating a trigger message. The CI events now do that for us. This
meant that parsing the branch name failed, and resulted in the empty
string.

Fix this by using already-parsed branch name from the CI event instead
of trying to parse it again.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-22 12:56:23 +02:00
Lars Wirzenius 70f1ba7de5 chore: prepare release 0.9.0
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-21 10:19:56 +02:00
Lars Wirzenius 54c3459aa4 feat: add repository alias to event queue
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-20 12:41:41 +02:00
Lars Wirzenius b707c812df feat: add count of recent failures to front page
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-20 12:17:54 +02:00
Lars Wirzenius d5f6520486 feat: add run counts for all repositories together to report
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-20 11:36:39 +02:00
Lars Wirzenius 331caf78ff feat: add per-repository run count (total, failed) to report
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-20 11:32:57 +02:00
Lars Wirzenius b8027442ca fix: add a time stamp to RSS feed items
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-14 07:11:37 +02:00
Lars Wirzenius d7bccbfc0b fix: don't run tidy in subplot
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 18:13:21 +02:00
Lars Wirzenius 4caab2118c feat: generate RSS feeds for all, failed CI runs
This is primarily a stop-gap measure so I can have some notification
of CI runs, without having to manually check the report page. It may
also be useful for other people, who do not use Radicle themselves,
but are interested in CI runs on Radicle nodes.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 18:06:19 +02:00
Lars Wirzenius dc61d2bbbb doc: reorder sections so that the more general case comes first
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 09:47:26 +02:00
Lars Wirzenius 92cbd0c730 doc: add goals to CI architecture doc
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 09:47:26 +02:00
Lars Wirzenius b199175ac0 doc: generate sample adapter messages with code
This removes the need to update them manually. Now they'll always be
up to date.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 09:47:26 +02:00
Lars Wirzenius 7ddb877a15 doc: format with Subplot, improve text
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 09:47:22 +02:00
Lars Wirzenius 43c076c22e feat: generate sample adapter messages with cibtool
This is meant only for generating documentation, but might be useful
for others.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-13 09:45:29 +02:00
Lars Wirzenius c377b05a97 test: add step to install an adapter
This is simpler to understand, for most readers of the subplot, than
installing a file and adjusting its Unix permissions with chmod.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 16:50:19 +02:00
Lars Wirzenius d76efe4007 test: have just one step to install all of CI broker
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 16:04:59 +02:00
Lars Wirzenius 842ce74f02 test: add new step "then stdout is empty"
This is a little clearer than 'then stdout is exactly ""'

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 15:34:32 +02:00
Lars Wirzenius 768b966890 fix: drop non-logging stderr output outside tests
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 15:11:37 +02:00
Lars Wirzenius 1e1587e820 feat: make sensitive_env optional in the configuration file
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 14:48:50 +02:00
Lars Wirzenius f3b83c30d8 feat: drop timeoutcmd.rs log messages to trace level
Also move them to logger.rs.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 09:19:45 +02:00
Lars Wirzenius 5aea92df7b feat: make it harder to leak sensitive environment variables
Add a new type for sensitive values, for `sensitive_envs`.

The CI broker can define sensitive environment variables for each
adapter, in the `sensitive_envs` field in the configuration file. The
idea is the this can be used to configure, say, API keys that the CI
run needs. To avoid them leaking accidentally, the code tries to
prevent the sensitive value from being printed. However, there have
been case where this has failed to be done correctly. The problem is
that each place where the `Config` value is used needs to be careful
to avoid leaking sensitive data.

The new type provides the protection in a more concentrated location.
The value can be created via serde de-serialization, but can't be
printed using normal formatting, debug formatting, or serde
serialization. This doesn't protect against the sensitive value
leaking via a core dump, or direct memory reading.

None of this makes it a good idea to use environment variables to
convey sensitive values. They're a bad idea and one should find a
better way to achieve the same thing. However, as long as the CI
broker provides the `sensitive_envs` configuration variable the least
it can do is try to make it hard to accidentally leak the values.

This is not a breaking change.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-12 08:17:43 +02:00
Lars Wirzenius 26d55f0604 feat: add link to front page to the per-repository report page
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-08 14:53:24 +02:00
Lars Wirzenius 10b6e6da9e test: use more recent releases for upgrade testing
Each additional version makes the upgrade test take longer, and we
only really need to care about recent releases, I think.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-08 13:49:51 +02:00
Lars Wirzenius 869b451728 chore: prepare release 0.8.0
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-08 10:17:11 +02:00
Lars Wirzenius 55e1e7dba5 feat: add optional max_run_time field to configuration file
Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-11-06 18:52:30 +02:00
Lars Wirzenius 30435e2001 feat: use new timeout-command module to execute adapter
The time out is hard coded to be 1000 seconds in this commit. Making
it configurable will come later.
2024-11-06 17:34:17 +02:00
Lars Wirzenius 0e78c6a8d8 feat: add a "cibtool timeout" subcommand
This lets us use the previously added code to run a sub-process with a
time limit. This is only meant for experimentation.
2024-11-06 16:30:42 +02:00
Lars Wirzenius 68f392dd78 feat: add a module to run a sub-process with a time limit
Also, capture its stdout and stderr.
2024-11-06 16:23:51 +02:00
Lars Wirzenius 0f90e97a3e feat: notify page generation about changes to runs more often
Change src/adapter.rs to send notification via the run notification
channel when the adapter is spawned, and when the adapter sends a
"triggered" message.

Signed-off-by: Lars Wirzenius <liw@liw.fi>
2024-10-21 16:19:09 +03:00