logical replication & CDC
Postgres already writes every change to a log for crash recovery; CDC just reads that log, so you get a live feed of the whole database for free.
leads to safe cutoversWhat I'm exploring, and how one idea led into the next. Hover a node to see what it connects to; click to jump to the note.
things I've builtideasbuilding next
The model is the ML half; the rest is infra: deciding which requests run together, where their memory lives, and how fast tokens come back.
Continuous batching and a paged KV cache in PyTorch for a small 0.5–1B model, benchmarked on tokens/sec and time to first token against naive batching, then vLLM.
a fixed pool of KV blocks, handed out to requests and returned when they finish
The pool is allocated up front, so nothing gets allocated in the middle of a request.
from building ferryman & parallax
logical replication & CDCPostgres already writes every change to a log for crash recovery; CDC just reads that log, so you get a live feed of the whole database for free.
leads to safe cutovers
parallel snapshot backfillOne exported snapshot lets many workers copy a huge table at once and still agree on the exact moment they all copied.
leads to safe cutovers
safe cutoversSwitching databases is a few seconds of choreography: wait for lag to hit zero, bump sequences, flip, and keep replication running backwards in case you need to retreat.
checked first with shadow traffic
shadow trafficTest a new system on real production traffic without a single user ever seeing its answers.
leads to serving metricswhere the next project lives
prefix caching & cache-aware routingThousands of requests start with the same system prompt; send them to the server that already computed it and the first token shows up almost instantly.
leads to long context windows
long context windowsDouble the context and attention costs four times as much, so long context is really a memory problem wearing a model feature's clothes.
leads to the inference server
serving metricsAverages lie: users feel the p99, and real traffic shows up in random bursts, not a polite steady stream.
leads to the inference server