Commit Graph
78 Commits
Author SHA1 Message Date
Alexis Sellier 5fa677c5fb node: Fix worker fetch deadlock
It turns out it's possible to create a deadlock when using
`child.wait()` with `Stdio::piped()`, if we don't read the output while
the child process is running. This is because the pipe has an internal
kernel buffer that will eventually fill up, causing the child process to
block on writing to stdout and never exit.

To remedy this situation, we read the stdout in a thread while the child
process is running.
2023-05-23 16:12:52 +02:00
Alexis Sellier eef813fd46 node: Small simplification in wire protocol 2023-05-12 11:08:29 +02:00
Alexis Sellier 1d9ee81b6d node: Rename reactor module to io
The name was always confusing and didn't mean much in the contetx of the
service.
2023-05-08 19:31:20 +02:00
Alexis Sellier ca0abc4356 node: Keep track of connection attempts
When attempting to connect to a peer, and when succeeding, write the timestamp
to the database.

This will allow us to better select addresses and not retry peers to
soon.
2023-05-05 17:23:52 +02:00
Alexis Sellier 83547f9c8a node: Close worker streams on disconnect
This is in an attempt to fix the worker pool from being backed up.
2023-04-27 11:42:33 +02:00
Alexis Sellier ccbfe5b6fa node: Add warning if workers are all busy 2023-04-26 09:59:19 +02:00
Alexis Sellier 10fe2ae299 node: Add worker channel timeouts
It's currently possible for a channel to be blocked forever while
waiting for data, if the remote peer keeps the connection alive.
This can occupy a worker indefinitely.

With these timeouts, we ensure that workers exit if there is no
activity after a certain amount of time.
2023-04-26 09:59:19 +02:00
Alexis Sellier 5428420f27 node: Rework peer map to not panic
In the wire protocol, we often panic in case a peer is not found.
With this change, we instead log an error.
2023-04-26 09:59:19 +02:00
Alexis Sellier fd3a6583d0 node: Improve framing code
Thanks to @fintohaps for these suggestions.
2023-04-23 15:10:42 +02:00
Alexis Sellier 94bef61944 node: Return only remotes that were fetched from
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.

This can differ if some trusted peers were not available
on the remote node.

Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
2023-04-17 18:58:56 +02:00
Alexis Sellier 4b2bc365ef node: Better differentiate fetch results
We use an enum to separate initiator vs. responder results.
2023-04-17 16:18:34 +02:00
Alexis Sellier 51b372ecad node: Share channel type between modules
Shares the `Channels` type between the `wire::protocol` and
`worker` modules.
2023-04-17 15:56:24 +02:00
Alexis Sellier 18616a16bd node: Match error by value
There wasn't a good reason to use a reference. This also allows us
to explicitly drop the session.
2023-04-14 13:55:47 +02:00
Alexis Sellier 17651e259d node: Don't panic if peer disconnected 2023-04-14 13:55:32 +02:00
Alexis Sellier bda5d6f080 node: Tunnel git data without parsing packet-lines
Instead of parsing the worker requests and responses, which was causing issues
around packfile transmission, we use threads and pass the bytes
transparently.
2023-04-14 13:16:32 +02:00
Alexis Sellier 273ce2744a node: Add more information to worker result log 2023-04-14 13:15:52 +02:00
Alexis Sellier ea4294c79f node: More realistic concurrent fetch test
This implements a test which can optionally use a lot more data by
setting `RAD_TEST_SCALE` to a higher value.
2023-04-14 13:15:52 +02:00
Alexis Sellier 31647521fa node: Properly close worker channels
We need to send a 'close' event to the worker channel for it to exit
without an error.

Also updated and improved a bunch of the logs.
2023-04-14 13:15:52 +02:00
Alexis Sellier 89b9eb53b7 node: Implement protocol multiplexing
To improve the reliability and flexibility of the protocol, we introduce
multiplexing over peer connections. This involves a new `Frame` type
that carries a stream-id and payload.

Three stream types are made available:

1. Control
2. Gossip
3. Git

This change brings the following improvements:

* Removed need to queue fetch requests
* Removed need to queue gossip messages
* Removed need for `Fetch` and `FetchOk` messages
* Service doesn't need to know about inbound fetches
* Removed one round-trip for fetch negotiation
* Removed special `done` git packet
* Removed session logic and state around Git/Fetch protocols
* Removed code around upgrading/downgrading transport
* Worker in responder mode is able to process any number of fetches
* Connections support any number of concurrent fetches

We had to introduce a few extra things however to make it all work:

* A `VarInt` type for variable-length integers, since we want the frames
  to be lightweight.
* A custom "tunnel" implementation, since we couldn't use the existing
  one anymore.

Overall the change removes more complexity than it adds, while improving
the protocol along the way.
2023-04-10 14:41:55 +02:00
Alexis Sellier fe8953ab8b node: Don't penalize peer for protocol mismatch
Instead of returning a non-transient disconnect reason, return
a transient one, since protocol mismatches can happen.
2023-03-24 13:23:43 +01:00
xphoniex a49eec9892 node: Add eventing system to Handler and Service
Signed-off-by: xphoniex <dj.2dixx@gmail.com>
2023-03-16 22:26:57 +01:00
Alexis Sellier fdfe1d508b node: Use deserializer when decoding messages
For whatever reason, we weren't using the deserializer for message
decoding. This meant that if a message came in two pieces, it would
cause decoding problems, since the first piece would be discarded.
2023-03-16 15:26:13 +01:00
Alexis Sellier 17e3c975fb node: Better error message when protocol mismatch 2023-03-16 15:26:13 +01:00
Alexis Sellier 98ee9c46a9 node: Fix state mismatch between service and wire
Due to never removing peers from the peer map in the wire
protocol module, it was possible for there to be multiple
connected peers with the same Node ID. Some might be in a
"connected" state, while others might be "disconnected".

This caused a bug where a peer, although connected would
be considered disconnected by the wire module, because
when searching for a peer with that Node ID, the disconnected
one would return first.

To fix this, we make sure to remove peers from the map once
the connection is dropped.
2023-03-15 16:46:27 +01:00
Alexis Sellier 40c58b45ed node: Move routing to radicle crate 2023-03-15 12:59:49 +01:00
Alexis Sellier 94427ee22b node: Call service on failed connection attempt
Before this change, we were not calling back into the service due
to the missing `NodeId`.
2023-03-13 16:31:00 +01:00
Fintan Halpenny e408b0d62c node: use enum for fetch direction
A fetch is a client-server interaction and so there are two sides to
this interaction. These sides are generally referred to as receive --
the client side -- and upload -- the server side.

The `Fetch` type indicated whether it was doing a receive or upload by
using a field `initiated: bool`. However, this can be confusing when
you pair it with the `namespaces: Namespaces` field, since only the
receive side should be sending what refspecs it wants and the upload
side responds with those matching refspecs.

A better way to represent this is using an enum for which direction
the fetch is being considered, where the receive side contains the
Namespaces. To borrow from Noise, the variant names are 'Initiator'
for the client and 'Responder' for the server.

The resulting code then makes decisions based on what variant of the
enum was passed.

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-03-08 10:12:54 +00:00
Alexis Sellier 94899f7be2 node: Update localtime crate 2023-03-07 10:47:12 +01:00
Alexis Sellier a5ee494fda node: Correctly handle disconnect in wire protocol 2023-03-06 15:39:07 +01:00
Dr Maxim Orlovsky 158f1ff7fa Update to fixed io-reactor timer 2023-02-22 16:21:25 +01:00
Alexis Sellier a7d2fa95cd node: Remove redundant handshake
With `NoiseXK` handshake implemented, there is no longer a need to
have an `Initialize` message and an extra state in the session.
2023-02-16 15:10:31 +01:00
Alexis Sellier 8cbc3c2403 node: Move to NoiseXK
Move from NoiseNN with a custom authentication protocol, to NoiseXK.
Uses ECDH on the ed25519 curve via the `ec25519` library.
2023-02-15 22:05:04 +01:00
Alexis Sellier ee513ccf7e node: Complete the "relay" seed test
These are the final changes to get the "relay" scenario passing.
There are still some issues left, though it's done as far as the
test is concerned.
2023-02-14 20:31:40 +01:00
Alexis Sellier c138cd062c node: Give each peer its own inbox
Instead of a shared queue, each peer gets its own inbox.
2023-02-08 11:43:39 +01:00
Alexis Sellier 054655d79b node: Log read buffer if not empty 2023-02-07 21:13:29 +01:00
Alexis Sellier f66ecd4ece node: Replication fixes
* Read git request line before starting daemon, so that we don't leave
  git data in the socket read buffer in case of an error connecting to
  the daemon.
* Send a response on the fetch channel even if we disconnect the remote.
* Don't send `done` packet if there's an error; it won't be expected.
2023-02-07 21:04:13 +01:00
Alexis Sellier f5f36191d9 node: Fix some connection-related logic 2023-02-07 10:47:35 +01:00
Alexis Sellier 45e31a5192 Communicate fetch results through control socket
This allows a CLI to report to the user what fetch requests were
fulfilled, and which ones failed.
2023-02-07 10:46:12 +01:00
Fintan Halpenny 3f48c2c516 Rust 1.67
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.

[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html

Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
2023-02-04 19:30:29 +01:00
Alexis Sellier ac23f45d9b node: Cleanup working naming 2023-01-29 21:41:16 +01:00
Alexis Sellier b4f44f39f2 Update netservices dependency 2023-01-25 15:35:24 +01:00
Alexis Sellier 963ad3c3d7 Get rad clone working correctly 2023-01-25 11:35:20 +01:00
Alexis Sellier ef950d515a node: Fully integrate worker fetch into service 2023-01-25 10:21:57 +01:00
Alexis Sellier 3bc07195f5 Migrate to localtime from nakamoto
The only thing we needed nakamoto for was `LocalTime`.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 18:03:11 +01:00
Alexis Sellier d25c47ad0e node: Get rid of worker response channel 2023-01-18 17:45:46 +01:00
Alexis Sellier 7e55ed346a node: Update log target to match type name
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 17:45:46 +01:00
Alexis Sellier 067b532154 node: Get replication test to pass
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 17:45:46 +01:00
Alexis Sellier 723cdcf233 node: Update io-reactor
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 17:45:46 +01:00
Alexis Sellier f33e79f14b node: Fix redundant announcements
This change makes sure that nodes don't gossip announcements back to
the announcers.

Since the tests were relying on this mechanism to populate the routing
table, we had to change a few more things, such as adding the inventory
to the node's routing table.

Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-18 08:48:33 +01:00
Alexis Sellier 3bedb4438e node: Fix a few logging-related things
Signed-off-by: Alexis Sellier <self@cloudhead.io>
2023-01-17 11:01:36 +01:00