Skip to content

Commit 7a6e471

Browse files
NiKrauseNiKrause
committed
feat: add WebRTC Direct performance testing with @roamhq/wrtc
- Added webrtc-roamhq-wrtc implementation variant for performance testing - Updated runner to support WebRTC Direct multiaddr handling - Added Hetzner Cloud Terraform configuration as AWS alternative - Implemented listen address capture via /tmp/webrtc-listen-addrs.txt - Fixed libp2p v2.10.0 compatibility with exact dependency versions Performance Results (Hetzner Germany, 1 iteration): - Upload: WebRTC Direct 194 Mbps vs TCP 840 Mbps - Download: WebRTC Direct 186 Mbps vs TCP 1,236 Mbps - Latency: WebRTC Direct 323ms vs TCP 124ms WebRTC Direct achieves ~186-194 Mbps throughput with @roamhq/wrtc, demonstrating functional parity with expected WebRTC overhead (~4-6x slower than TCP). Related: libp2p/js-libp2p#3356, libp2p/js-libp2p#3034, libp2p/js-libp2p#3033
1 parent 70bb3fd commit 7a6e471

File tree

12 files changed

+2381
-26123
lines changed

12 files changed

+2381
-26123
lines changed

‎WEBRTC_PERF_IMPLEMENTATION.md‎

Lines changed: 291 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,291 @@
1+
# WebRTC Performance Testing Implementation
2+
3+
## Summary
4+
5+
This document describes the implementation of WebRTC Direct performance testing for libp2p/test-plans to validate PR #3356 (@roamhq/wrtc replacement).
6+
7+
## What Was Built
8+
9+
### 1. New Implementation: `webrtc-roamhq-wrtc`
10+
11+
**Location:** `/Users/nandi/test-plans/perf/impl/js-libp2p/webrtc-roamhq-wrtc/`
12+
13+
**Purpose:** Test WebRTC Direct transport using your PR #3356 which replaces node-datachannel with @roamhq/wrtc + stun.
14+
15+
**Key Features:**
16+
- Uses local build of your js-libp2p fork at `/Users/nandi/js-libp2p`
17+
- Dependencies point to local packages:
18+
- `libp2p`: `file:/Users/nandi/js-libp2p/packages/libp2p` (v3.1.2)
19+
- `@libp2p/webrtc`: `file:/Users/nandi/js-libp2p/packages/transport-webrtc` (v6.0.10)
20+
- Updated versions:
21+
- `@chainsafe/libp2p-noise`: v17.0.0
22+
- `@chainsafe/libp2p-yamux`: v7.0.4
23+
- `@multiformats/multiaddr`: v13.0.1
24+
25+
**Transport Support:**
26+
- ✅ WebRTC Direct (webrtc-direct)
27+
- ✅ Prints full multiaddr with certhash and peer ID to stderr
28+
- ✅ Client parses full multiaddr from `--server-address`
29+
30+
### 2. Runner Modifications
31+
32+
**File:** `/Users/nandi/test-plans/perf/runner/src/index.ts`
33+
34+
**Changes Made:**
35+
1. Added `getWebRTCDirectListenAddr()` function to extract multiaddr from server logs
36+
2. Modified `runBenchmarkAcrossVersions()` to:
37+
- Wait 2 seconds after server start for WebRTC Direct
38+
- Capture listen address from server log
39+
- Pass it to client
40+
3. Updated `runClient()` to use captured multiaddr for webrtc-direct transport
41+
4. Updated `ArgsRunBenchmark` interface with `serverListenAddr` field
42+
43+
**File:** `/Users/nandi/test-plans/perf/runner/versionsInput.json`
44+
45+
Added new version entry:
46+
```json
47+
{
48+
"id": "webrtc-roamhq-wrtc",
49+
"implementation": "js-libp2p",
50+
"transportStacks": ["webrtc-direct"]
51+
}
52+
```
53+
54+
### 3. Implementation Files Modified
55+
56+
**index.js** - Both variants updated to:
57+
- Support `--transport` flag (tcp, webrtc-direct)
58+
- Print `[LISTEN_ADDR] <multiaddr>` to stderr on server startup
59+
- Parse full multiaddr from `--server-address` for WebRTC Direct
60+
- Maintain stdout exclusively for JSON perf measurements
61+
62+
**package.json** - Updated dependencies to match PR versions
63+
64+
## How It Works
65+
66+
### Server Flow
67+
1. Server starts with `--run-server --server-address 0.0.0.0:4001 --transport webrtc-direct`
68+
2. libp2p binds to UDP port 4001 with WebRTC Direct
69+
3. Server prints to stderr: `[LISTEN_ADDR] /ip4/0.0.0.0/udp/4001/webrtc-direct/certhash/uEi.../p2p/12D3...`
70+
4. Server keeps running, waiting for connections
71+
72+
### Client Flow
73+
1. Runner extracts listen multiaddr from server.log
74+
2. Client receives full multiaddr: `/ip4/<server-ip>/udp/4001/webrtc-direct/certhash/.../p2p/...`
75+
3. Client dials server using this multiaddr
76+
4. Perf measurements stream to stdout as JSON
77+
78+
### Data Flow
79+
```
80+
Server (stderr) → server.log → Runner (grep) → Client (--server-address)
81+
Server/Client (stdout) → JSON metrics → Runner → benchmark-results.json
82+
```
83+
84+
## Running Performance Tests
85+
86+
### Prerequisites
87+
1. AWS credentials configured
88+
2. Terraform 1.5.4+
89+
3. Node.js 18+
90+
4. SSH key pair (or run `make ssh-keygen` in terraform directory)
91+
92+
### Step 1: Provision Infrastructure
93+
94+
```bash
95+
cd /Users/nandi/test-plans/perf/terraform/configs/local
96+
97+
# Generate SSH keys if needed
98+
make ssh-keygen
99+
make ssh-add
100+
101+
# Provision AWS EC2 instances
102+
terraform init
103+
terraform apply
104+
105+
# Capture instance IPs
106+
export CLIENT_IP=$(terraform output -raw client_ip)
107+
export SERVER_IP=$(terraform output -raw server_ip)
108+
```
109+
110+
### Step 2: Build Implementation
111+
112+
The runner automatically builds implementations, but you can test manually:
113+
114+
```bash
115+
cd /Users/nandi/test-plans/perf/impl/js-libp2p/webrtc-roamhq-wrtc
116+
npm install # Already done locally
117+
```
118+
119+
### Step 3: Run Performance Tests
120+
121+
```bash
122+
cd /Users/nandi/test-plans/perf/runner
123+
124+
# Install runner dependencies
125+
npm ci
126+
127+
# Run tests (10 iterations, ~20 seconds per test)
128+
npm run start -- \
129+
--client-public-ip $CLIENT_IP \
130+
--server-public-ip $SERVER_IP \
131+
--test-filter js-libp2p
132+
133+
# Or run in testing mode (1 iteration, 5 seconds per test)
134+
npm run start -- \
135+
--client-public-ip $CLIENT_IP \
136+
--server-public-ip $SERVER_IP \
137+
--test-filter js-libp2p \
138+
--testing
139+
```
140+
141+
### Step 4: Collect Results
142+
143+
Results are saved to: `/Users/nandi/test-plans/perf/runner/benchmark-results.json`
144+
145+
### Step 5: Cleanup
146+
147+
```bash
148+
cd /Users/nandi/test-plans/perf/terraform/configs/local
149+
terraform destroy
150+
```
151+
152+
## Test Outputs
153+
154+
### Benchmarks Run
155+
1. **throughput/upload** - Upload speed (bit/s)
156+
2. **throughput/download** - Download speed (bit/s)
157+
3. **Connection establishment + 1 byte round trip latencies** - Latency (s)
158+
159+
### Expected Results Format
160+
161+
```json
162+
{
163+
"benchmarks": [
164+
{
165+
"name": "throughput/upload",
166+
"unit": "bit/s",
167+
"results": [
168+
{
169+
"implementation": "js-libp2p",
170+
"version": "webrtc-roamhq-wrtc",
171+
"transportStack": "webrtc-direct",
172+
"result": [
173+
{"type": "intermediary", "timeSeconds": 1.0, "uploadBytes": 12500000, "downloadBytes": 0},
174+
...
175+
]
176+
}
177+
]
178+
}
179+
]
180+
}
181+
```
182+
183+
## Known Issues
184+
185+
### @roamhq/wrtc Segfault
186+
- **Status:** Known issue with @roamhq/wrtc
187+
- **Impact:** Segmentation fault during process cleanup (after tests complete)
188+
- **Workaround:** Does not affect test execution or measurements
189+
- **Tests Passing:** All 33 functional tests pass before cleanup
190+
191+
### Local vs Remote Build
192+
- **Important:** The implementation uses local file references to your PR
193+
- **For CI/CD:** Replace file references with commit hash:
194+
```json
195+
"@libp2p/webrtc": "github:NiKrause/js-libp2p#bb4ee1749814d0a9bdf44e2b9fcacf3ec6ad71dd"
196+
```
197+
198+
## Troubleshooting
199+
200+
### Server doesn't print listen address
201+
- Check: `ssh ec2-user@$SERVER_IP 'cat server.log'`
202+
- Expected: Should see `[LISTEN_ADDR] /ip4/...`
203+
204+
### Client fails with "NoValidAddressesError"
205+
- Cause: Runner didn't capture listen address correctly
206+
- Fix: Check server.log exists and contains LISTEN_ADDR line
207+
208+
### Build failures on EC2
209+
- Cause: Missing Node.js or npm on EC2 instances
210+
- Fix: Ensure EC2 instances have Node.js 18+ (usually pre-configured in terraform)
211+
212+
### Dependencies not found
213+
- Cause: Local js-libp2p not built
214+
- Fix: `cd /Users/nandi/js-libp2p && npm run build`
215+
216+
## Next Steps
217+
218+
1. ✅ **Implementation Complete** - All code ready
219+
2. 🔄 **Run Tests** - Execute performance suite on AWS
220+
3. 📊 **Analyze Results** - Compare throughput metrics
221+
4. 📝 **Report Findings** - Post to issues #3033 and PR #3356
222+
223+
### Analysis Checklist
224+
- [ ] Upload throughput (bit/s)
225+
- [ ] Download throughput (bit/s)
226+
- [ ] Connection latency (s)
227+
- [ ] Stability (any crashes beyond known segfault?)
228+
- [ ] Memory usage patterns
229+
- [ ] Comparison with baseline (if node-datachannel variant tested)
230+
231+
### Reporting Template
232+
233+
Post to **Issue #3033** and **PR #3356**:
234+
235+
```markdown
236+
## WebRTC Direct Performance Results
237+
238+
### Test Environment
239+
- Implementation: webrtc-roamhq-wrtc (PR #3356)
240+
- Transport: WebRTC Direct
241+
- Iterations: 10
242+
- Duration: 20 seconds per test
243+
244+
### Results
245+
246+
#### Upload Throughput
247+
- Mean: X Mbit/s
248+
- Median: X Mbit/s
249+
- Std Dev: X Mbit/s
250+
251+
#### Download Throughput
252+
- Mean: X Mbit/s
253+
- Median: X Mbit/s
254+
- Std Dev: X Mbit/s
255+
256+
#### Connection Latency
257+
- Mean: X ms
258+
- Median: X ms
259+
- P95: X ms
260+
261+
### Observations
262+
- [Stability notes]
263+
- [Performance observations]
264+
- [Comparison with node-datachannel if available]
265+
266+
### Dashboard
267+
[Link to Observable dashboard visualization]
268+
269+
### Recommendation
270+
[Proceed with @roamhq/wrtc / Keep node-datachannel / Further testing needed]
271+
```
272+
273+
## Files Modified
274+
275+
### Created
276+
- `/Users/nandi/test-plans/perf/impl/js-libp2p/webrtc-roamhq-wrtc/` (entire directory)
277+
278+
### Modified
279+
- `/Users/nandi/test-plans/perf/runner/src/index.ts`
280+
- `/Users/nandi/test-plans/perf/runner/versionsInput.json`
281+
282+
### External Dependencies
283+
- `/Users/nandi/js-libp2p/` (PR #3356 local build)
284+
285+
## References
286+
287+
- **Issue #3033:** WebRTC streaming performance
288+
- **Issue #3034:** Evaluate @roamhq/wrtc vs node-datachannel
289+
- **PR #3356:** Implementation using @roamhq/wrtc
290+
- **Test Plans Repo:** https://github.com/libp2p/test-plans
291+
- **Dashboard:** https://observablehq.com/@libp2p-workspace/performance-dashboard

‎perf/impl/js-libp2p/webrtc-roamhq-wrtc/Makefile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
DOCKER_IMAGE := node:22-alpine
2+
DOCKER_RUN := docker run --rm -v "$(shell pwd)":/usr/src/myapp -w /usr/src/myapp $(DOCKER_IMAGE)
3+
4+
all: perf
5+
6+
perf:
7+
@echo "Dependencies pre-installed via GitHub reference"
8+
9+
clean:
10+
rm -rf node_modules
11+
12+
.PHONY: all clean perf

0 commit comments

Comments
 (0)