Order-book collector

The result

Two hours on AWS, both venues at once: Polymarket's full universe from London (c7i.xlarge, 4 vCPU, 8 GB) and Kalshi's whole demo universe from Virginia. The hour-long run the day before held 550k Polymarket books at 42k updates/s.

The pipeline

One engine, five stages. Only the first two know which venue they are talking to; that seam is the ExchangeAdapter trait (Ext 3).

1. 01DiscoveryLists every live market at start and every 10 minutes, and follows the venue's listing and resolution events in between. New markets join; resolved ones are dropped.
2. 02Shards and socketsMarkets are split into shards of 2,500 tokens. Each shard gets two websockets on different edge servers (Ext 1), so losing one socket or one edge loses nothing.
3. 03DecodeTokio workers read the sockets and turn the venue's JSON into a common event: a price, a size and a timestamp.
4. 04Core threadsEach core thread owns whole markets. It applies events to flat sorted books, chooses which replica is the source, and checks each book against the venue's hash.
5. 05WriterEncodes updates and snapshots with the template's SBE helpers into 64 MiB chunks. Each chunk is written as it fills, then the file is rotated off the hot path.

When something goes wrong

Each case below has a test that injects the fault against a mock venue, and most were also seen live.

What the challenge asked for

Each claim, with the command that reproduces it: GRADING.md. The design choices and the alternatives they beat: DECISIONS.md.

Every run, by host

Five stages, in the order they ran. Each move came from what the run before it showed. Every output file that was closed replayed with 0 mismatches.

Raw logs for every run are in bench/; the full story, turn by turn, is the engineering log.

How it got here

The first build was correct but small. Each run at a bigger scale or a harder fault found the next limit; each limit got a fix and a measurement. The charts follow the same metric across the runs, labelled with the fixes that landed before each one.

The fixes

In the order they were found. Each line says what hit the limit, how it was found, and what the fix moved.

1. #1
   Laptop link behind a VPN, 3.3 Mbit/s

   Run 4 collapsed into a reconnect storm. Moved to a cloud host near London; added a global outage backoff.

   wire p99: minutes → 173 ms
2. #2
   Coverage stuck at 47.5k of 131k tokens

   Coverage grew one shard per 30 s, then stopped. The open limiter now reads the current gap; books that never arrive are re-asked.

   47.5k → 96k, later 100% placed
3. #3
   Socket cap and rate-limited lookups

   Cloud 1 left 196k tokens unplaced and 102 lookups got 429s. The cap is sized to the universe; lookups are batched with backoff.

   250k → 459k placed, 0 429s
4. #4
   Timers starved by inbound messages

   Cloud 2 printed one status line in 5 minutes. Timers run first; inbound is taken in batches of 256.

   p99 71 s → 524 ms
5. #5
   The hash check took 45% of the core

   A perf profile showed allocation per level and a full rehash per candidate price. Hash once, copy the partial state, format without allocating.

   18.9k → 32.3k updates/s
6. #6
   One core thread at 99.9%

   Per-thread CPU from /proc. K core threads now each own whole markets.

   99.9% → 50–90% per core
7. #7
   Ext 2 over-tuned

   Soak 1 made 400k resubscribes in 90 minutes. Quiet detection became rate-aware, and the wait doubles on each strike.

   resubscribes cut to the truly quiet tokens
8. #8
   Hashing standbys and a full lag walk

   A whole-process profile. Only the source replica is hashed; the lag check keeps an incremental candidate list.

   core CPU 50–60% → 31–38%
9. #9
   Rotation fsync stalled every reader

   Profile 2 had 330 s and 1,153 s blocked in rotation windows. A finisher thread fsyncs and renames off the writer.

   blocked time at rotation → 0
10. #10
    Hot-shard splitting caused a reconnect loop

    The A/B showed 11–19× the disconnects: the venue closes a socket whose assets have all resolved. Resolved tokens are dropped; splitting is opt-in.

    loop gone
11. #11
    Memory at 2.8–3.3 GB

    An offline bench found 7.6 KB of heap per token. mimalloc, flat sorted books, inline replicas, one symbol per token, lazy hash state, FxHash.

    RSS 3.0 → 1.73 GB
12. #12
    Disk stalls could reach the books

    The 2.5-hour soak had 8 writer waits of 1–1.5 s. A byte-bounded 256 MiB writer queue absorbs them.

    8 waits → 0
13. #13
    The receive time included TLS and parsing

    --kernel-ts reads the kernel's arrival time (SO_TIMESTAMPNS) and reports the host's own delay.

    host delay measured: p50 ≤ 32 µs on AWS
14. #14
    --pin started one core thread instead of two

    Found in the gate log. CPUs are counted once, before pinning.

    2 core threads, 281k / 280k tokens
15. #15
    One tokio worker for about 450 TLS sockets

    Host delay was 4× worse in gate 2. A 5-minute A/B of three layouts; io workers now default to at least 2.

    host p99 8–16 ms → 4–8 ms
16. #16
    A kill -9 without the supervisor lost the file

    Chunks were written at the end. Each chunk is now written in place as it fills.

    data kept after kill at 120 s: 0 → 119.8 s

Ruled out by measurement: the TCP source port (Ext 4, Experiment 3), hyperthread siblings for the core threads, and the ~1% of tokens with no book (the venue has none).

How correctness is proven

The output carries its own proof. A snapshot of a side plus every update after it must equal the next snapshot of that side, exactly. verify rebuilds every book from the updates and checks this for every snapshot in a file; any mismatch exits non-zero.

The method

The decisions that shape it

Each design choice, the alternative it beat, and the evidence. All 21 are in DECISIONS.md.

Built, measured, and kept off

Four ideas were built in full and A/B tested. The data did not justify them as defaults, so each stays as an opt-in flag and the null result is written down.

Where the time goes

Three numbers, each with a different clock problem. Only the last one is time spent inside the collector.

The four experiments

Experiment 4, round by round

Every round, table and raw log: LATENCY.md.

Kalshi, the challenge's venue

The challenge names Kalshi and allows Polymarket as the fallback. --venue kalshi runs the same core, socket actor, writer and verify on a Kalshi adapter (Ext 3). The adapter is the only part that knows Kalshi exists.

How it was tested

Details: KALSHI.md.

Run it

Needs stable Rust. One command builds, replays the committed sample, runs the offline tests and collects two minutes of live Polymarket data:

./demo.sh

Polymarket, by hand

cargo build --release --bins
./target/release/collector --max-markets 2000 --duration-secs 120 > ob.sbe
./target/release/verify ob.sbe

The output goes to stdout by default and status lines to stderr. --out ob.sbe writes the file directly; rotation needs that form.

Kalshi

Put a key in a gitignored .env (KALSHI_API_KEY_ID, KALSHI_PRIVATE_KEY_PATH), then:

./demo.sh kalshi                # production
KALSHI_DEMO=1 ./demo.sh kalshi  # demo environment
./demo.sh both                  # both venues at once

Elsewhere

HOW_TO_RUN.md GRADING.md Source