Commit Graph
27 Commits
Author SHA1 Message Date
cloudhead 1d167581f2 cob: Introduce xyz.radicle.id COB
Introduces a new COB type to store repository identity documents.

The reason for this change is to:

1. Simplify the code, as the identity document logic resembled a COB,
   yet it had custom logic. This allows existing COB code to be used for
   identities.
2. Make identity document update logic more flexible, since COB actions
   can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
   cob`.
4. Unify the concept of an identity change proposal, with regular identity
   changes. This means we can remove the `id.proposal` COB in favor of
   using the `id` COB itself.

Notes
-----

* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
  actions.
* The action that contains new document versions is called `revision`,
  just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
  COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
  Identity COB that contains the latest *accepted* revision of the
  document.
* All commands for managing identities have been folded into `rad id`.
  Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
  commands.
* When a revision has enough signatures, it is automatically adopted as
  the current identity, there is no longer the need for a "commit"
  action.
* When an identity revision is proposed, and the current identity has a
  threshold of `1`, that identity is automatically accepted due to the
  above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
  COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
  to point to (it would have to point to itself, which is impossible),
  we've made the resource id optional for COBs.
2023-10-12 17:32:33 +02:00
Alexis Sellier bb099faa4f node: Give useful thread names
For the purposes of debugging running nodes, it's useful to know which
threads are doing what. This allows tools like `htop` to reveal which
threads are running.
2023-06-20 11:18:14 +02:00
Alexis Sellier 5fa677c5fb node: Fix worker fetch deadlock
It turns out it's possible to create a deadlock when using
`child.wait()` with `Stdio::piped()`, if we don't read the output while
the child process is running. This is because the pipe has an internal
kernel buffer that will eventually fill up, causing the child process to
block on writing to stdout and never exit.

To remedy this situation, we read the stdout in a thread while the child
process is running.
2023-05-23 16:12:52 +02:00
Alexis Sellier 7037b52474 node: Read with timeout from worker channel
This change ensures that if we're waiting on a Git request packetline
that never arrives, we time out.
2023-05-11 14:52:55 +02:00
Fintan Halpenny 8508eab8fa radicle: update radicle-git-ext
This change updates to the latest radicle-git-ext, which required a
few changes within the radicle family of crates.

One set of changes is that the radicle-git-ext crate subsumes
git-commit, git-trailers, and git-ref-format -- so all those imports
go through radicle-git-ext.

The commit code requires that parent `Oid`s point to commits, which
made some tests fail. One of the reasons for these failures is that
the patch tests used fabricated `Oid`s and so the verification would
fail. This is fixed by allowing the test context code to create
`PatchRequest`s that create valid commits.
The other reason for failure was that the identity `Action` passed an
`Oid` that pointed to a blob. This is fixed by removing that code and
making a note of it in documentation.

The final major change was moving the Refspec type into git-ref-format
and removing the AsRefspecs trait. The trait was not required and
could its usage could be replaced by simpler code.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-05-08 10:36:15 +01:00
Alexis Sellier 0886ab0a3c node: Get the failing fetch tests passing
We fix the failing fetch tests by:

1. Not verifying our own refs, unless we're cloning, since we're
   otherwise not fetching our own refs.
2. Always force-fetching `sigrefs` from remotes into the staging copy.
3. Making sure that sigref updates are fast-forward before transfering
   the remote into the production copy.
2023-05-03 16:16:34 +02:00
Fintan Halpenny 3bfea69642 node: validate production in debug mode
Ensure that the production repository is validated after fetching and
performing deletions. This is to highlight any inconsistencies that
may show up between fetching and deleting references.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-04-28 14:26:47 +02:00
Fintan Halpenny 694bd345d9 node: use ls-remote when in fetching mode
Given the scenario of 3 peers, A, B, and C each having a project R. If
peer A has B's fork, but C does not, then when peer A attempts to
fetch from C the fetch will be rejected since Git will fail when a
refspec that is asked for does not exist on the remote side.

To avoid this, the fetch logic is changed so that the client first calls
ls-remote to the remote side, which results in the references that the
remote has which the client is interested in. The follow-up fetch then
uses these refs as the refspecs -- which should succeed unless there
was a race for a deletion on the remote side.

The rest of the verification logic stays the same, so the storage
should still be in a working state before transferring from the staged
repository to the production repository.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-04-28 14:26:47 +02:00
Alexis Sellier 83547f9c8a node: Close worker streams on disconnect
This is in an attempt to fix the worker pool from being backed up.
2023-04-27 11:42:33 +02:00
Alexis Sellier 10fe2ae299 node: Add worker channel timeouts
It's currently possible for a channel to be blocked forever while
waiting for data, if the remote peer keeps the connection alive.
This can occupy a worker indefinitely.

With these timeouts, we ensure that workers exit if there is no
activity after a certain amount of time.
2023-04-26 09:59:19 +02:00
Alexis Sellier fd3a6583d0 node: Improve framing code
Thanks to @fintohaps for these suggestions.
2023-04-23 15:10:42 +02:00
Alexis Sellier 2598886018 node: Delete unsigned refs on fetch 2023-04-19 11:32:46 +02:00
Alexis Sellier 0d98b84861 node: Skip trusted peers that aren't local 2023-04-18 14:49:13 +02:00
Alexis Sellier 94bef61944 node: Return only remotes that were fetched from
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.

This can differ if some trusted peers were not available
on the remote node.

Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
2023-04-17 18:58:56 +02:00
Alexis Sellier 51b372ecad node: Share channel type between modules
Shares the `Channels` type between the `wire::protocol` and
`worker` modules.
2023-04-17 15:56:24 +02:00
Alexis Sellier d0e24bd0ce node: Simplify worker data piping 2023-04-17 15:55:31 +02:00
Alexis Sellier 0e7ffcc916 node: Properly validate signed refs on fetch
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
2023-04-15 00:17:17 +02:00
Alexis Sellier 9729a23559 node: Use StagedRepository to know what to do
To know whether or not we're cloning, use `StagedRepository` instead
of checking manually.
2023-04-14 13:56:52 +02:00
Alexis Sellier bda5d6f080 node: Tunnel git data without parsing packet-lines
Instead of parsing the worker requests and responses, which was causing issues
around packfile transmission, we use threads and pass the bytes
transparently.
2023-04-14 13:16:32 +02:00
Alexis Sellier 31647521fa node: Properly close worker channels
We need to send a 'close' event to the worker channel for it to exit
without an error.

Also updated and improved a bunch of the logs.
2023-04-14 13:15:52 +02:00
Alexis Sellier c8e548fb1d node: Make sure we have all refs in staging copy
Previous to this change, we just created a clone, which was missing
all the refs.
2023-04-12 11:43:43 +02:00
Alexis Sellier eeec2008e8 node: Allow most refs to be force-updated
Previously, during the "transfer" stage of a fetch, we used `force:
false`, which didn't allow refs to be force-updated.

This change makes it possible to fetch force-updated refs, but ensures
that the sigrefs branch is never force-updated.
2023-04-12 11:09:05 +02:00
Alexis Sellier 89b9eb53b7 node: Implement protocol multiplexing
To improve the reliability and flexibility of the protocol, we introduce
multiplexing over peer connections. This involves a new `Frame` type
that carries a stream-id and payload.

Three stream types are made available:

1. Control
2. Gossip
3. Git

This change brings the following improvements:

* Removed need to queue fetch requests
* Removed need to queue gossip messages
* Removed need for `Fetch` and `FetchOk` messages
* Service doesn't need to know about inbound fetches
* Removed one round-trip for fetch negotiation
* Removed special `done` git packet
* Removed session logic and state around Git/Fetch protocols
* Removed code around upgrading/downgrading transport
* Worker in responder mode is able to process any number of fetches
* Connections support any number of concurrent fetches

We had to introduce a few extra things however to make it all work:

* A `VarInt` type for variable-length integers, since we want the frames
  to be lightweight.
* A custom "tunnel" implementation, since we couldn't use the existing
  one anymore.

Overall the change removes more complexity than it adds, while improving
the protocol along the way.
2023-04-10 14:41:55 +02:00
Alexis Sellier ed4c266a51 node: Don't prune owned refs
Owned refs were being pruned on fetch.
2023-04-03 22:09:41 +02:00
Fintan Halpenny c618c3e443 node: remove Namespaces::One
The One variant caused many paint points for fetching logic, where it
was not necessary. It was only constructed in one place, which could
be replaced by using the variant that holds a set of keys.

Remove the Namespaces::One variant and rename Namespaces::Many to
Namespaces::Trusted.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:23:32 +01:00
Fintan Halpenny adf6312a55 node: fetch trusted peers on clone
When a repository does not yet exist during a fetch, i.e. a clone,
then only delegates are being fetched.

Augment Namespaces::Many to hold the trusted and optional delegate peers
separately. Doing so allows the construction of the variant without
the repository existing, but tracking relationships existing.

This variant can then be used to build refspecs for both the trusted
peers and delegates when doing a cloning fetch.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:23:31 +01:00
Fintan Halpenny db0ceddf6b node: multistep fetching logic
The previous fetching logic had some pitfalls regarding its logic:

* Fetched 'rad' references could end up not being verifiable and leaving
  the storage in a corrupt state
* Forced pushes would be prohibited, since they would be rejected by
  the client side due to the refspecs being used.

The new approach is to stage a fetch in 2 steps. The first step is to
fetch the relevant `rad/id` and `rad/sigrefs` for verification. For
all remotes that are verified, fetch all references listed in their
respective `rad/sigrefs`, using the `+` marker in the refspec,
allowing to update for any force pushes -- note that this becomes a
safe operation since the `sigrefs` are signed by the remote that
created them.

To prevent unverifiable 'rad' references from polluting storage, all
fetch negotiations are done in a staging repository set up in a
temporary directory. All successful fetches are then transferred,
using the 'file://' protocol, from the temporary directory into the
radicle storage.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:19:36 +01:00