relayforge

RelayForge

A multi-link SRT failover controller with live telemetry. It polls a MediaMTX instance, scores each publish link’s health with a deterministic state machine, picks the single ACTIVE link, and (optionally) drives OBS so the broadcast follows the healthy link — then streams that health to a browser Mission Control dashboard.

Status: failover core + telemetry + dashboard + a self-contained local demo. This is the v0/tracer slice of a larger plan (live recording + richer UI to come).

Live demo (no install)

▶ Play the recorded failover in your browser — real telemetry captured from the loopback demo, replayed through the same dashboard renderer. No clone, no MediaMTX, no setup. Pick a recording (failover / recover / degraded), scrub the timeline, watch a link die and the controller switch over.

Architecture

flowchart LR
  A[linkA] -->|SRT| MTX[MediaMTX]
  B[linkB] -->|SRT| MTX
  MTX -->|"API :9997 — paths + srtconns"| CTRL["controller.py<br/>compute_state · decide"]
  CTRL -->|"SSE /events · /snapshot"| DASH[Mission Control dashboard]
  CTRL -->|"one snapshot / poll"| JSONL[(JSONL)]
  JSONL -.->|replay| REPLAY[replay.html · zero-install]
  CTRL -->|"obs_apply"| OBS[OBS scene switch]

Senders are synthetic / local ffmpeg in the demos; real phone/cellular senders are hardware (out of scope here). The controller’s own work is the health state machine, the decision, the telemetry, and the OBS integration. The ~3 s freeze path is shown via the synthetic source; the OBS program switch is shown below.

What this is / what it isn’t

Deterministic, threshold-based. No AI/ML — compute_state() is pure threshold logic: GOOD / DEGRADED / DEAD with hysteresis (FREEZE_POLLS, OFFLINE_POLLS, DEGRADE_POLLS) and a switch COOLDOWN.
Failover (proven locally). Killing one of several SRT publishers into the real MediaMTX + controller stack triggers an automatic switch to the next healthy link. In the loopback demo below this is measured at ~1 poll (~1–2 s) for a clean disconnect (the SRT socket closes). A real link death where packets stop but the socket stays open is caught by the freeze rule (FREEZE_POLLS=3) in ~3 s. Both paths are deterministic. The broadcast (OBS output) itself does not drop — only the dead link’s video is interrupted during the switch; this is not hitless.
Telemetry is new work. The original controller emitted a terminal TUI + logfile only. The HTTP/SSE telemetry server, the JSONL snapshot stream, and the dashboard are added here. state / bitrate / freeze come from /v3/paths/list bytesReceived deltas; rtt / loss are shown when MediaMTX reports them on /v3/srtconns.
Scope. Full-mesh / single-host — individual to small scale, not an SFU.

Heritage / contribution boundary

The health state machine is the original car-stream controller (akagifreeez/mediamtx-failover-controller). The transmission pieces of the wider stack are upstream OSS — the SRT sender app (CarStream, built on StreamPack) and the IPv6 ingest recipe (srtla-ipv6-bonding, whose core is the belabox SRT fork / irl-srt-server / srtla_send). This repo’s own work = the integration, the telemetry layer, the state machine, and the dashboard.

Run the demo (no cellular, no real lines, no OBS needed)

Requires python 3.9+, ffmpeg, and a mediamtx binary.

cd demo
python run_demo.py --mediamtx /path/to/mediamtx --auto    # scripted: kills linkA, observe, exit
# or, hands-on:
python run_demo.py --mediamtx /path/to/mediamtx           # then kill linkA's ffmpeg yourself

Open http://127.0.0.1:8080 to watch link cards flip GOOD → DEAD and the ACTIVE marker move when a link dies. Every poll is also appended to demo/demo-run.jsonl.

Mission Control — before / after a link death

Both links healthy, linkA ACTIVE:

before

linkA killed → DEAD, controller fails over to linkB (see the switch log):

after

See demo/RESULTS.md for the measured timing.

The broadcast follows the link (OBS)

With OBS running (obs-websocket enabled), the controller enables only the ACTIVE link’s source, so the program output itself switches on failover. Here linkA (blue) is live; after linkA is killed, OBS switches to linkB (green):

OBS program before/after failover — blue linkA then green linkB

Verified both visually and via obs-websocket scene-item state (linkA:on,linkB:off → linkA:off,linkB:on). Run it:

pip install obsws-python
cd demo && python run_obs_demo.py --mediamtx /path/to/mediamtx --auto

Controller (standalone)

python controller.py --no-supervise --headless \
  --api http://127.0.0.1:9997/v3 --http-port 8080 --jsonl run.jsonl

GET / — the Mission Control dashboard
GET /events — Server-Sent Events; one full snapshot per poll
GET /snapshot — the current snapshot as JSON

All paths/ports/thresholds are configurable via CLI or RELAYFORGE_* env vars (controller.py --help). The demo MediaMTX config (demo/mediamtx-loopback.yml) is environment-specific by design.

Snapshot schema (the contract)

{"ts": 0.0, "active": "linkA", "api_ok": true,
 "links": [{"name":"linkA","state":"GOOD","bitrate_kbps":1550.0,
            "freeze":0,"rtt_ms":null,"loss_pct":null,"readers":0,"uptime_s":7}]}

Tests

python -m unittest discover -s tests -t .

16 deterministic tests cover compute_state() and decide() — GOOD/DEGRADED/DEAD with hysteresis, failover, failover-to-degraded, no-flap on a degraded primary, cooldown recovery to the primary, and stale-link exclusion. No network, MediaMTX, or OBS required; they run in CI on every push.

License

MIT — see LICENSE.

This site is open source. Improve this page.