libdpf/docs/STREAM_ARRAY_GAP_REPORT.md

Stream-array / network layer report

Interactive MPC/FSS that is prep plus message rounds runs on net::async_stream_array (in-process memory, unix sockets, TCP mux, parallel TCP, SCTP) through async_round_sink, compose_async, and the N-party runner. Every decision the network layer makes is an explicit, per-object setting; the library reads no environment variables.

Configuration

Setting	Type / where
Wire policy: window, max frame, chunk size, coalescing (frames and bytes), inbox compaction, socket options	net::wire_policy (net/policy.hpp), passed to every backend constructor; set_window_bytes at run time
Socket options: TCP_NODELAY, quickack, keepalive (idle/interval/count), SO_SNDBUF/SO_RCVBUF, SCTP_NODELAY	net::socket_options inside wire_policy
Setup deadlines: join, connect, accept, handshake, drain	net::deadlines
Lanes and framing	drive_options::n_lanes (0 = 8, lanes_one_per_round), framing_mode {automatic, always, never}
Wait budgets	drive_options::wait_timeout per round wait, edge_timeout[edge], round_timeout[round]; the clock restarts whenever an instance advances, so a long healthy run never times out
Kernels	drive_options::workers (compute pool) and pump (the sink's io_context, serviced while a kernel runs)
Harness	app::run_config: transport, host, lanes, framing, instances, wire policy, pipeline credit, wait budget, compute threads, warmup, trials, CPU pins, deadlines. from_env() (DPF_<KEY>) and apply_args() (--key=value) share one parser; unknown keys and bad values throw

pipeline_credit only lets a session submit rounds whose bytes do not depend on the peer, and only while that edge's window has room (RoundSink::can_send_ahead). Compose plans are fully dependent; overlap them with instances.

Backends

Backend	Notes
async_dual_memory_hub / make_async_dual_memory_stream_pair	One io_context per side; per-writer window; closing a side still delivers accepted writes, then EOF
async_mux_stream_array	One TCP socket; writes are chunked and round-robined across lanes so a small write is not queued behind a large one; up to coalesce_frames / coalesce_bytes per syscall; reads land directly in a waiting reader's buffer
basic_async_parallel_stream_array (TCP and unix)	One socket per lane with its own strand, queue, and window (lane_window_bytes); accept_parallel_tcp / connect_parallel_tcp use one port and a lane-index handshake
async_sctp_stream_array (Linux + libsctp)	Lane i = SCTP stream i; per-stream chunked round-robin sends straight from the owned payload; window; sctp_available() checks support up front
mux_stream_array (synchronous)	Same wire format as the async mux (interoperates with it); a poll-based pump drains reads while writing, so two large cross flushes do not deadlock

All backends report stream_stats (wire bytes including headers, payload bytes, frames, write calls, buffered/unread bytes, last activity, error). close() aborts; destruction is graceful and drains accepted writes.

Round sink

async_round_sink(streams, slots, instances, sink_options):

• Plan-shape hello (rounds, slot widths, lanes, instances, framing, epoch); a mismatch fails both ends with a message naming the field.
• Framed lanes carry {round, nbytes}; partial instance prefixes are ready as they land. Unframed lanes carry one round each.
• flush_round waits while the lane's window is full, up to drain_timeout.
• sink_options::reconnect supplies a replacement link after a transport error (for example party_session::reconnector(peer)). Both ends exchange what they received and resend the rest, so a plan resumes mid-run (sink_stats::resumes, resent_bytes).
• Separate out and in links (async_round_sink(out, in, ...)) for the RSS ring.

Sessions and multi-party runs

Surface	Where
party_session: static host:port table (any start order), connect with retry to a deadline, per-edge transport (mux / parallel / SCTP) and wire policy, handshake that names mismatches, per-edge stats and epochs, reconnect / reconnector	net/party_session.hpp
Dealer link in its own failure domain with its own epoch, policy, and io_context; dealer_session serves several parties	net/party_session.hpp
run_parties (2 or 3 parties, dealer and RSS-ring edges) on every transport; run_two_party / run_three_party	party_runner.hpp, launch.hpp
One process per party: parse_node_args (--party=i --peers=host:port,... plus any run_config key) and run_node	party_runner.hpp, examples/protocol/party_node.cpp
Prep over a dealer session on the configured transport	session::ship_prep(demand, cfg)

Link security

Party links (party to party, and the dealer) run TLS 1.3 through OpenSSL on every socket edge unless encryption=off. Each party has a raw Ed25519 key (identity = FILE, made with examples/tools/dpf_keygen.cpp; 44 characters of base64, like a WireGuard key). There are no certificate files and no CA: the certificate TLS needs is generated in memory from the key, and peers check the key inside it.

Configured	Effect
no keys at all	links are encrypted to a fresh key per run; nobody is authenticated; each side logs security.no_identity and security.unauthenticated
peer.N = KEY on party M	M authenticates N; N still does not authenticate M unless it holds M's key
a held key that does not match	the link fails at setup with both keys in the message
dealer_key = KEY	the party authenticates the dealer (the dealer uses peer.N for parties)
SCTP edge while encrypted	refused at setup (no TLS over SCTP streams); encryption=off allows it

The session hello (party id, epoch, transport, lanes) travels inside TLS and carries whether the sender authenticated the receiver, so every link.up record shows encryption=TLSv1.3/<cipher>, auth=key|none, peer_key, and peer_verified_us.

Client links (dpf/net/client_link.hpp): a client_listener presents server_cert/server_key (PEM, CA-issued), or server_identity, or, with neither, the built-in development certificate. connect_server always verifies the server unless client_verify=off: a pinned key (client_pin), a CA chain for the host name (client_ca = FILE|system, client_server_name), or, when neither is configured, the development certificate. That certificate's private key is public, so the default pairing works out of the box and both ends log that it provides no security. A server may check client keys (server_client_pin, with client_identity on the client).

All of it is run_config keys, as flags, DPF_* variables, or a file of key = value lines (--config=FILE, DPF_CONFIG; relative paths resolve against the file):

# p0.conf
identity   = p0.key
peer.1     = file:p1.pub
dealer_key = 8s3b...=
transport  = mux

Harness

app::exercise_parties(plans, inputs, kernels, ex, cfg) runs each party's plan from fresh inputs for warmup untimed and trials timed repetitions and records the median. experiment writes config.csv (every run_config setting), trials.csv, and wire.csv (party 0's link counters) next to the existing tables. experiment_bench takes every setting as a flag.

Remaining

1. QUIC: fits the async_stream_array interface; not implemented.
2. Synchronous entry points (tcp_pair, tcp_pair_mux, join_*_tcp_mesh, trio) go through the same peer_security / party_session framework as async edges (TLS 1.3 by default). Sync mux is a blocking face over async_stream_array.
3. SCTP links cannot be encrypted.
4. The synchronous sctp_stream_array face still throws; use async_sctp_stream_array.
5. Non-Linux SCTP is not supported; constructors throw and sctp_available() is false.

Verification

net_control_test (27 tests) covers the controls above: config parsing, framing modes, hello mismatches, partial prefixes, window gating, per-edge and per-round budgets, a healthy run longer than its wait budget, mux round-robin and graceful close, per-lane parallel windows, sync/async mux interop, connect deadlines, static tables, transport mismatch, parallel and SCTP edges, mid-plan reconnect, dealer epochs, every transport through run_parties, a 3-party dealer and ring plan, separate processes from a static table, harness trials and CSVs, and prep over a dealer session. share_runtime_test (91 tests) covers the rest of the stack. security_test covers key files, TLS in every authentication combination, a recording relay that sees only ciphertext, wrong keys, mixed encryption settings, the SCTP refusal, missing-key logging, dealer keys, every client-link mode (development default, pins, CA chain and host name, verify off, client keys), config files, and two processes that authenticate each other from key files.