140 lines
8.9 KiB
Markdown
140 lines
8.9 KiB
Markdown
|
# Stream-array / network layer report
|
||
|
|
|
||
|
|
Interactive MPC/FSS that is prep plus message rounds runs on
|
||
|
|
`net::async_stream_array` (in-process memory, unix sockets, TCP mux, parallel
|
||
|
|
TCP, SCTP) through `async_round_sink`, `compose_async`, and the N-party runner.
|
||
|
|
Every decision the network layer makes is an explicit, per-object setting; the
|
||
|
|
library reads no environment variables.
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
| Setting | Type / where |
|
||
|
|
|---------|--------------|
|
||
|
|
| Wire policy: window, max frame, chunk size, coalescing (frames and bytes), inbox compaction, socket options | `net::wire_policy` (`net/policy.hpp`), passed to every backend constructor; `set_window_bytes` at run time |
|
||
|
|
| Socket options: `TCP_NODELAY`, quickack, keepalive (idle/interval/count), `SO_SNDBUF`/`SO_RCVBUF`, `SCTP_NODELAY` | `net::socket_options` inside `wire_policy` |
|
||
|
|
| Setup deadlines: join, connect, accept, handshake, drain | `net::deadlines` |
|
||
|
|
| Lanes and framing | `drive_options::n_lanes` (`0` = 8, `lanes_one_per_round`), `framing_mode {automatic, always, never}` |
|
||
|
|
| Wait budgets | `drive_options::wait_timeout` per round wait, `edge_timeout[edge]`, `round_timeout[round]`; the clock restarts whenever an instance advances, so a long healthy run never times out |
|
||
|
|
| Kernels | `drive_options::workers` (compute pool) and `pump` (the sink's `io_context`, serviced while a kernel runs) |
|
||
|
|
| Harness | `app::run_config`: transport, host, lanes, framing, instances, wire policy, pipeline credit, wait budget, compute threads, warmup, trials, CPU pins, deadlines. `from_env()` (`DPF_<KEY>`) and `apply_args()` (`--key=value`) share one parser; unknown keys and bad values throw |
|
||
|
|
|
||
|
|
`pipeline_credit` only lets a session submit rounds whose bytes do not depend on
|
||
|
|
the peer, and only while that edge's window has room (`RoundSink::can_send_ahead`).
|
||
|
|
Compose plans are fully dependent; overlap them with `instances`.
|
||
|
|
|
||
|
|
## Backends
|
||
|
|
|
||
|
|
| Backend | Notes |
|
||
|
|
|---------|-------|
|
||
|
|
| `async_dual_memory_hub` / `make_async_dual_memory_stream_pair` | One `io_context` per side; per-writer window; closing a side still delivers accepted writes, then EOF |
|
||
|
|
| `async_mux_stream_array` | One TCP socket; writes are chunked and round-robined across lanes so a small write is not queued behind a large one; up to `coalesce_frames` / `coalesce_bytes` per syscall; reads land directly in a waiting reader's buffer |
|
||
|
|
| `basic_async_parallel_stream_array` (TCP and unix) | One socket per lane with its own strand, queue, and window (`lane_window_bytes`); `accept_parallel_tcp` / `connect_parallel_tcp` use one port and a lane-index handshake |
|
||
|
|
| `async_sctp_stream_array` (Linux + libsctp) | Lane `i` = SCTP stream `i`; per-stream chunked round-robin sends straight from the owned payload; window; `sctp_available()` checks support up front |
|
||
|
|
| `mux_stream_array` (synchronous) | Same wire format as the async mux (interoperates with it); a poll-based pump drains reads while writing, so two large cross flushes do not deadlock |
|
||
|
|
|
||
|
|
All backends report `stream_stats` (wire bytes including headers, payload
|
||
|
|
bytes, frames, write calls, buffered/unread bytes, last activity, error).
|
||
|
|
`close()` aborts; destruction is graceful and drains accepted writes.
|
||
|
|
|
||
|
|
## Round sink
|
||
|
|
|
||
|
|
`async_round_sink(streams, slots, instances, sink_options)`:
|
||
|
|
|
||
|
|
* Plan-shape hello (rounds, slot widths, lanes, instances, framing, epoch); a
|
||
|
|
mismatch fails both ends with a message naming the field.
|
||
|
|
* Framed lanes carry `{round, nbytes}`; partial instance prefixes are ready as
|
||
|
|
they land. Unframed lanes carry one round each.
|
||
|
|
* `flush_round` waits while the lane's window is full, up to `drain_timeout`.
|
||
|
|
* `sink_options::reconnect` supplies a replacement link after a transport error
|
||
|
|
(for example `party_session::reconnector(peer)`). Both ends exchange what
|
||
|
|
they received and resend the rest, so a plan resumes mid-run
|
||
|
|
(`sink_stats::resumes`, `resent_bytes`).
|
||
|
|
* Separate out and in links (`async_round_sink(out, in, ...)`) for the RSS ring.
|
||
|
|
|
||
|
|
## Sessions and multi-party runs
|
||
|
|
|
||
|
|
| Surface | Where |
|
||
|
|
|---------|-------|
|
||
|
|
| `party_session`: static `host:port` table (any start order), connect with retry to a deadline, per-edge transport (mux / parallel / SCTP) and wire policy, handshake that names mismatches, per-edge stats and epochs, `reconnect` / `reconnector` | `net/party_session.hpp` |
|
||
|
|
| Dealer link in its own failure domain with its own epoch, policy, and `io_context`; `dealer_session` serves several parties | `net/party_session.hpp` |
|
||
|
|
| `run_parties` (2 or 3 parties, dealer and RSS-ring edges) on every transport; `run_two_party` / `run_three_party` | `party_runner.hpp`, `launch.hpp` |
|
||
|
|
| One process per party: `parse_node_args` (`--party=i --peers=host:port,...` plus any `run_config` key) and `run_node` | `party_runner.hpp`, `examples/protocol/party_node.cpp` |
|
||
|
|
| Prep over a dealer session on the configured transport | `session::ship_prep(demand, cfg)` |
|
||
|
|
|
||
|
|
## Link security
|
||
|
|
|
||
|
|
Party links (party to party, and the dealer) run TLS 1.3 through OpenSSL on
|
||
|
|
every socket edge unless `encryption=off`. Each party has a raw Ed25519 key
|
||
|
|
(`identity = FILE`, made with `examples/tools/dpf_keygen.cpp`; 44 characters of
|
||
|
|
base64, like a WireGuard key). There are no certificate files and no CA: the
|
||
|
|
certificate TLS needs is generated in memory from the key, and peers check the
|
||
|
|
key inside it.
|
||
|
|
|
||
|
|
| Configured | Effect |
|
||
|
|
|------------|--------|
|
||
|
|
| no keys at all | links are encrypted to a fresh key per run; nobody is authenticated; each side logs `security.no_identity` and `security.unauthenticated` |
|
||
|
|
| `peer.N = KEY` on party M | M authenticates N; N still does not authenticate M unless it holds M's key |
|
||
|
|
| a held key that does not match | the link fails at setup with both keys in the message |
|
||
|
|
| `dealer_key = KEY` | the party authenticates the dealer (the dealer uses `peer.N` for parties) |
|
||
|
|
| SCTP edge while encrypted | refused at setup (no TLS over SCTP streams); `encryption=off` allows it |
|
||
|
|
|
||
|
|
The session hello (party id, epoch, transport, lanes) travels inside TLS and
|
||
|
|
carries whether the sender authenticated the receiver, so every `link.up` record
|
||
|
|
shows `encryption=TLSv1.3/<cipher>`, `auth=key|none`, `peer_key`, and
|
||
|
|
`peer_verified_us`.
|
||
|
|
|
||
|
|
Client links (`dpf/net/client_link.hpp`): a `client_listener` presents
|
||
|
|
`server_cert`/`server_key` (PEM, CA-issued), or `server_identity`, or, with
|
||
|
|
neither, the built-in development certificate. `connect_server` always verifies
|
||
|
|
the server unless `client_verify=off`: a pinned key (`client_pin`), a CA chain
|
||
|
|
for the host name (`client_ca = FILE|system`, `client_server_name`), or, when
|
||
|
|
neither is configured, the development certificate. That certificate's private
|
||
|
|
key is public, so the default pairing works out of the box and both ends log
|
||
|
|
that it provides no security. A server may check client keys
|
||
|
|
(`server_client_pin`, with `client_identity` on the client).
|
||
|
|
|
||
|
|
All of it is `run_config` keys, as flags, `DPF_*` variables, or a file of
|
||
|
|
`key = value` lines (`--config=FILE`, `DPF_CONFIG`; relative paths resolve
|
||
|
|
against the file):
|
||
|
|
|
||
|
|
```text
|
||
|
|
# p0.conf
|
||
|
|
identity = p0.key
|
||
|
|
peer.1 = file:p1.pub
|
||
|
|
dealer_key = 8s3b...=
|
||
|
|
transport = mux
|
||
|
|
```
|
||
|
|
|
||
|
|
## Harness
|
||
|
|
|
||
|
|
`app::exercise_parties(plans, inputs, kernels, ex, cfg)` runs each party's plan
|
||
|
|
from fresh inputs for `warmup` untimed and `trials` timed repetitions and
|
||
|
|
records the median. `experiment` writes `config.csv` (every `run_config`
|
||
|
|
setting), `trials.csv`, and `wire.csv` (party 0's link counters) next to the
|
||
|
|
existing tables. `experiment_bench` takes every setting as a flag.
|
||
|
|
|
||
|
|
## Remaining
|
||
|
|
|
||
|
|
1. QUIC: fits the `async_stream_array` interface; not implemented.
|
||
|
|
2. Synchronous entry points (`tcp_pair`, `tcp_pair_mux`, `join_*_tcp_mesh`, `trio`) go through the same `peer_security` / `party_session` framework as async edges (TLS 1.3 by default). Sync mux is a blocking face over `async_stream_array`.
|
||
|
|
3. SCTP links cannot be encrypted.
|
||
|
|
4. The synchronous `sctp_stream_array` face still throws; use `async_sctp_stream_array`.
|
||
|
|
5. Non-Linux SCTP is not supported; constructors throw and `sctp_available()` is false.
|
||
|
|
|
||
|
|
## Verification
|
||
|
|
|
||
|
|
`net_control_test` (27 tests) covers the controls above: config parsing, framing
|
||
|
|
modes, hello mismatches, partial prefixes, window gating, per-edge and per-round
|
||
|
|
budgets, a healthy run longer than its wait budget, mux round-robin and graceful
|
||
|
|
close, per-lane parallel windows, sync/async mux interop, connect deadlines,
|
||
|
|
static tables, transport mismatch, parallel and SCTP edges, mid-plan reconnect,
|
||
|
|
dealer epochs, every transport through `run_parties`, a 3-party dealer and ring
|
||
|
|
plan, separate processes from a static table, harness trials and CSVs, and prep
|
||
|
|
over a dealer session. `share_runtime_test` (91 tests) covers the rest of the stack.
|
||
|
|
`security_test` covers key files, TLS in every authentication combination, a
|
||
|
|
recording relay that sees only ciphertext, wrong keys, mixed encryption
|
||
|
|
settings, the SCTP refusal, missing-key logging, dealer keys, every client-link
|
||
|
|
mode (development default, pins, CA chain and host name, verify off, client
|
||
|
|
keys), config files, and two processes that authenticate each other from key
|
||
|
|
files.
|