Commit Graph
43 Commits
Author SHA1 Message Date
Alexis Sellier 94bef61944 node: Return only remotes that were fetched from
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.

This can differ if some trusted peers were not available
on the remote node.

Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
2023-04-17 18:58:56 +02:00
Alexis Sellier 0e7ffcc916 node: Properly validate signed refs on fetch
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
2023-04-15 00:17:17 +02:00
Alexis Sellier a05de1fbf3 node: Test fetch with large repository
This is for stress-testing the replication.
2023-04-14 13:15:52 +02:00
Alexis Sellier f4257eb953 node: Add setting for limiting fetch concurrency
Can be passed via `--limit-fetch-concurrency <num>`.
2023-04-14 13:15:52 +02:00
Alexis Sellier ea4294c79f node: More realistic concurrent fetch test
This implements a test which can optionally use a lot more data by
setting `RAD_TEST_SCALE` to a higher value.
2023-04-14 13:15:52 +02:00
Alexis Sellier 31647521fa node: Properly close worker channels
We need to send a 'close' event to the worker channel for it to exit
without an error.

Also updated and improved a bunch of the logs.
2023-04-14 13:15:52 +02:00
Alexis Sellier ed4c266a51 node: Don't prune owned refs
Owned refs were being pruned on fetch.
2023-04-03 22:09:41 +02:00
Fintan Halpenny adf6312a55 node: fetch trusted peers on clone
When a repository does not yet exist during a fetch, i.e. a clone,
then only delegates are being fetched.

Augment Namespaces::Many to hold the trusted and optional delegate peers
separately. Doing so allows the construction of the variant without
the repository existing, but tracking relationships existing.

This variant can then be used to build refspecs for both the trusted
peers and delegates when doing a cloning fetch.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:23:31 +01:00
Fintan Halpenny f78e010c55 node: e2e test simulating a peer migrating their key
Previously, if a peer created a project which was replicated and then
moved to another setup, ultimately deleting that project, then they
would not be able to fetch it again since their remote namespace would
be excluded from the fetch.

This test confirms that a peer can remove their project and clone it
again from a peer that has replicated the project. This is simulated
by directly removing the project directory in the test and fetching
the project from the other peer.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:19:37 +01:00
Fintan Halpenny 6e08f0be59 node: enable test_fetch_trusted_remotes
With the latest fetch updates, the test for fetching trusted remotes
can be enabled.

The caveat is that its expected behaviour is to fetch the delegate
upon a clone and fetch all trusted peers on a subsequent
fetch. Fetching all trusted peers is left as a follow-up task.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:19:36 +01:00
Fintan Halpenny db0ceddf6b node: multistep fetching logic
The previous fetching logic had some pitfalls regarding its logic:

* Fetched 'rad' references could end up not being verifiable and leaving
  the storage in a corrupt state
* Forced pushes would be prohibited, since they would be rejected by
  the client side due to the refspecs being used.

The new approach is to stage a fetch in 2 steps. The first step is to
fetch the relevant `rad/id` and `rad/sigrefs` for verification. For
all remotes that are verified, fetch all references listed in their
respective `rad/sigrefs`, using the `+` marker in the refspec,
allowing to update for any force pushes -- note that this becomes a
safe operation since the `sigrefs` are signed by the remote that
created them.

To prevent unverifiable 'rad' references from polluting storage, all
fetch negotiations are done in a staging repository set up in a
temporary directory. All successful fetches are then transferred,
using the 'file://' protocol, from the temporary directory into the
radicle storage.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-31 12:19:36 +01:00
Fintan Halpenny a629f91b1e node: set identity head after fetch
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-27 10:13:03 +01:00
Alexis Sellier 05d7090326 node: Remove Handle::inventory method 2023-03-15 12:59:49 +01:00
Fintan Halpenny 821a6b6c08 node: test do not fetch from owned refs
To ensure the behaviour of ignoring references that are owned by the
local peer a test case is added.

The case has two peers: alice and bob, where alice creates a new
repository. bob fetches to be up-to-date with alice. alice, in turn,
creates an issue and fetches from bob -- which should be successful.

If the refspec:

    ^refs/namespaces/<alice>/*

was not in place, her fetch from bob would fail since the signed
references would not be a fast-forward.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-08 11:33:42 +00:00
Fintan Halpenny 7fddc70a57 node: test fetch from trusted
Adds a test for fetching from trusted delegates.

Currently marked as failing. When the fetch is initiated from a client
that does not have the repository then it will fetch all namespaces,
instead of the trusted set.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-08 11:33:42 +00:00
Fintan Halpenny 3b015a9325 radicle: accept Scope as track_repo argument
The method `track_repo` only accepted the RID as the argument for
tracking.

Instead, allow passing Scope as an argument for tracking a
repository.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-08 11:29:34 +00:00
Fintan Halpenny c70dc71b18 node: improve seeds information
Previously, the seeds information would only consist of the connected
NodeIds.

Improve on this by adding any disconnected and fetching seeds as
well. A seed can either be disconnected, connected, or fetching --
where fetching implies connected.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-02-27 17:48:11 +01:00
Alexis Sellier 950fae209d Make ReadStorage more usable
Previously, we were not able to access a `ReadRepository` instance
through `ReadStorage`. With this change, we are able to.
2023-02-15 22:08:20 +01:00
Alexis Sellier f5f36191d9 node: Fix some connection-related logic 2023-02-07 10:47:35 +01:00
Alexis Sellier 2649e9c6fe cli: Add rad-clone test 2023-02-07 10:46:12 +01:00
Alexis Sellier b542ddcd6c node: Create shared test environment
We move and combine some of the test code, so that the CLI tests and
node tests use shared code.
2023-02-07 10:46:12 +01:00
Alexis Sellier 45e31a5192 Communicate fetch results through control socket
This allows a CLI to report to the user what fetch requests were
fulfilled, and which ones failed.
2023-02-07 10:46:12 +01:00
Fintan Halpenny 3f48c2c516 Rust 1.67
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.

[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-02-04 19:30:29 +01:00
Alexis Sellier 33d8c35189 node: Move some code to Runtime::run 2023-01-29 17:39:46 +01:00
Alexis Sellier 23cf44f4c7 node: Rename client module to runtime 2023-01-29 17:33:07 +01:00
Alexis Sellier 02be334144 node: Use git-daemon as backend 2023-01-29 17:26:01 +01:00
Alexis Sellier 30e9d6103c Cleanup clone code
Remove a bunch of different instances of it.
2023-01-25 17:51:49 +01:00
Alexis Sellier fea14ff7b3 Update clone functionality for new replication 2023-01-25 10:21:57 +01:00
Alexis Sellier ef950d515a node: Fully integrate worker fetch into service 2023-01-25 10:21:57 +01:00
Alexis Sellier ec2e7d7ee1 node: Refactor fetch response code
Only user-requested fetches require a response.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-25 10:21:57 +01:00
Alexis Sellier 3aff91f9b7 Correctly initialize home in Profile::init 2023-01-22 13:39:48 +01:00
Alexis Sellier d25c47ad0e node: Get rid of worker response channel 2023-01-18 17:45:46 +01:00
Alexis Sellier 067b532154 node: Get replication test to pass
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 17:45:46 +01:00
Alexis Sellier 723cdcf233 node: Update io-reactor
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 17:45:46 +01:00
Alexis Sellier b9484b6832 node: Implement graceful shutdown
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
2023-01-18 17:44:16 +01:00
Alexis Sellier f33e79f14b node: Fix redundant announcements
This change makes sure that nodes don't gossip announcements back to
the announcers.

Since the tests were relying on this mechanism to populate the routing
table, we had to change a few more things, such as adding the inventory
to the node's routing table.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 08:48:33 +01:00
Alexis Sellier 3d04849b7b node: Properly parse replication data
Fix all of the `FIXME` in the worker.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 12:59:00 +01:00
Alexis Sellier df9346fefb node: Fix E2E tests
The tests were flaky because the routes are sampled at a moment
when gossip could still be happening. Instead, we check the
local inventory directly to build the expected routing table.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 12:59:00 +01:00
Alexis Sellier be9def8b11 Rename profile::Paths to profile::Home
This makes the intent clearer.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 11:01:36 +01:00
Alexis Sellier d31968509d Speed up node E2E tests
By allowing the node to run using a `MockSigner`, we speed up tests
significantly, since PBKDF2 doesn't need to run.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 11:01:36 +01:00
Alexis Sellier 741f7c9651 node: Rename Transport to Wire
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 11:01:36 +01:00
Alexis SellierandDr. Maxim Orlovsky 63e7248847 node: Complete Noise handshake integration
Co-authored-by: Dr. Maxim Orlovsky <dr.orlovsky@gmail.com>
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 11:01:36 +01:00
Alexis Sellier 438ae84304 node: Implement E2E node test
There are few unrelated changes in this commit that I wasn't able to
split out, mostly due to laziness, so I'll list them here:

* The node startup logic is moved out of the main function, and into a
  dedicated functionin `client.rs`.
* There is a tentative worker implementation that is untested, and
  handles git replication both ways. It's likely lacking in various ways
  but I wanted to get it in there anyway.
* `Address` was moved from `radicle-node` to `radicle`, because it's
  needed to send `Connect` commands from the node handle.
* The `netservices` dependency was updated.
* Finally, there is a first end-to-end node test, which currently
  doesn't use NoiseXK, but uses the new reactor in `netservices`.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-12 20:34:31 +01:00