An engineering log of building a Threads marketing agent, split into two parts.

• Part 1: The data-acquisition layer — a crawler designed around an anti-bot threat model (TLS fingerprint · CDP detection)

• Part 2 (this post): The generation pipeline — iterative content-model design, scheduled publishing, performance attribution

TL;DR

Part 1 secured posts from arbitrary sites into a single normalized schema ({title, body, comments[], images[]}). Part 2 covers three design decisions that wire it into an operable publish-and-measure pipeline.

1. Content model — evolved iteratively from a single content_mode (different criteria conflated on one axis) into four orthogonal axes (hook × persona × format × topic). The "4 axes" weren't designed up front; they're the result of resolving coupling / separation-of-concerns problems one at a time.

2. Scheduled publishing — rejected SaaS (Buffer etc.) on a build-vs-buy basis and implemented it with a SQLite queue + a 1-minute cron worker. Idempotency via atomic claim.

3. Attribution — measured the clicks and conversions Threads doesn't give you, in three layers (engagement → click → conversion). The hard part wasn't visualization but enforcing double-counting prevention, empty states, and the bot filter at the data-contract level.

Scope note: this is currently a measurement-infrastructure-first phase. It's a record of system design and problem-solving; business outcomes (ER/CVR/ROAS) require accumulated samples and are deliberately not claimed here (see §4).

1. Content model — iterative evolution toward orthogonal axes

The current generation skill defines a post as the product of four axes.

post = hook_engine × persona × format × (topic_tag*)   + voice (community vernacular, enforced)

• hook_engine — the hook angle (why you stop): hot_take, money_truth, regret_story, versus, rank_bait …

• persona — speaker/target: first_car, young_budget / purist, tuner, data_geek …

• format — body skeleton (the "vessel"): list_drip, confession, fill_in_blank, one_line_thread …

• topic_tag — topic domain: EV, depreciation, options, tuning …

This structure wasn't designed in one shot. It went through five turning points, each a classic software-design problem (separation of concerns, orthogonality, scope, a missing abstraction).

Turning point 0 — the root of a wrong abstraction: identity

The initial identity was a "latest-trend car curator." No amount of tone rules stopped the output from regressing to a "tidy news report." The cause wasn't tone — it was the higher abstraction, identity.

A curator organizes information; an operator designs reactions. Platform algorithms and communities respond not to organized information but to emotion and engagement. If the identity is "curator," output structurally converges to "read-and-done." Tone is a dependent variable of identity — without changing identity, no tone rule has force.

The structural flaws of "recency" dependence were big too. Material exhausts fast, differentiation drops to zero, and you end up chained to press releases (de facto PR work). The actual high performers were evergreen hooks (first-car regret, sucker options, "for that money…").

So I redefined the identity as a "community operator who designs reactions." Crucially, this didn't discard information — it kept information as the trust base layer. Too much info is boring; too much provocation is backlash. The target is the middle.

In parallel I enforced separation of concerns between SOUL (identity / invariant constraints) and SKILL (execution procedure). At the time, image policy, output format, comment composition, and body length were duplicated across both docs, causing real conflicts (comment composition: fixed vs dynamic; length: 6–9 lines vs 5–7 sentences). The rule is simple: numbers/format/procedure live in SKILL as the single source; SOUL holds only "the why and the ceilings." Single source of truth.

Turning point 1 — anti-pattern diagnosis: four criteria conflated on one axis

The initial four content_modes mixed different classification criteria on a single axis.

The result was classification collision: one post qualified for multiple modes at once (especially debate vs buyer). Worse, the "speaker (who's talking)" axis was missing entirely, so I couldn't regenerate the same car with just a different speaker. A textbook anti-pattern of cramming multiple concerns into one enum.

Turning point 2 — orthogonal decomposition: enum → product structure

I decomposed the conflated axis into three orthogonal axes (hook_engine × persona × topic_tag). Because the axes are independent, combinatorial variety is secured as a product. debate_thread was strengthened into hot_take; money_truth / regret_story / versus were added; persona was promoted to a first-class input (6 buyer types + 6 enthusiast types); and comments shifted from a fixed 3-persona set to dynamic mapping, where a speaker opposed to the body's axis pushes back.

Hidden coupling — schema migration. The publish-ledger recording script's validate() allowed only the old 4 modes, so generating with a new engine killed the record with SystemExit. Generation worked, but the input to the learning loop (anti-repeat / performance weighting) was severed. I fully replaced the vocabulary with v2 and normalized the past ledger via an idempotent migration to restore data consistency. A textbook case of a model change entailing a data-layer migration.

Turning point 3 — scope decision: introducing then retracting auto-discovery (Auto-Pick)

On top of the 3 axes I added Auto-Pick ("simple input → bot auto-selects the axes"). It worked, but I rejected it in operation.

Auto-discovering the material too increases topic dispersion. A marketing channel must select vetted posts, and that judgment is human intuition > bot heuristics. So I fixed the scope to "a human picks the source URL" (same as Part 1's Principle 2) and removed auto-discovery. The bot is responsible only for transformation and variety — a decision binding the scope of automation to the scale of operation.

Turning point 4 — discovering the missing abstraction: format (the body vessel)

Even with 3 axes, the body kept regressing to article-speak. Digging into why: hook_engine decides only "why you click (the first line / hook)", and there was no axis to set the body skeleton. So hook_engine took on that responsibility too and fell into information-listing. The fix was a fourth axis, format, to enforce the body skeleton.

Key: exclude the info-summary form (data_brief) from the default and allow it only on explicit request. The default is one of 5 community vessels. The same material becomes a new post by switching the vessel (sales figures → a list_drip, not a table).

voice hard constraints (reject on violation)

Even with the right axes, article-speak sentences mean failure, so the body voice has enforced rules.

1. Number cap — outside data_brief, ≤2 key numbers per body (rate-of-change / multi-model number dumps auto-rejected)

2. First sentence is situation/emotion/provocation — never open with data

3. No links in the body — sources go in comment footnotes only (reason in §2)

4. No source-community exposure — banned phrasing like "I saw on Bobaedream…"; rewrite in first person

Part 1's comment N<3 DROP gate operates here too.

Design retrospective: the 4 axes aren't a top-down design but an emergent structure from sequentially resolving separation of concerns (TP 1–2) → scope (3) → a missing abstraction (4). Each turn began with the observation "without this, it keeps falling into the same failure mode." A content model is a refactoring target, just like code.

2. Scheduled publishing — build vs buy, and an idempotent worker

Decision: rejected publishing SaaS (Buffer etc.). With a Discord bot, VPS, and SQLite already in place and simple requirements (slot scheduling / auto-publish / Discord integration), a self-built SQLite queue + 1-minute cron worker won on both coupling and cost.

Architecture

Discord: a human drops a source URL
   → generation skill: body + 3 comments + images
   → clarify buttons: [pick topic] [time: 14:23 / 19:47 / 🚀now]
   → threads_enqueue.py: INSERT into publish_queue (status=pending, scheduled_for)
        ┄┄┄ (async boundary) ┄┄┄
   → cron(1m) threads_publish_worker.py:
        atomically claim rows where scheduled_for ≤ now → 'publishing'
        → Threads Graph API publish (carousel ≤10)
        → update published (thread_url, published_at) + Discord notification

Design points:

• Idempotency — the worker first locks a due row to publishing (atomic UPDATE) before processing. No double-publish even on 1-minute re-entry.

• Slot-recommendation UX — so a human doesn't type a time, offer two slots within KST 10:00–24:00 + a "now" three-way choice.

• Rate limit — Threads API allows 250/account/24h. The queue drain rate is capped to that.

The real reason for the no-link-in-body rule (coupled with §1 voice #3): the performance-collection cron matches metrics to a post by comparing the published body and the DB-stored body character-by-character. A link in the body breaks that match. So the body is pure text and inbound links go entirely into comments (footnotes). It looks like a constraint, but it's intentional coupling for §3 attribution accuracy.

3. Attribution dashboard — the data contract is the design

The hard part wasn't visualization; it was enforcing measurement integrity at the data-contract level.

3.1 Why measure it ourselves — the 3-layer measurement boundary

Threads insights provide only impressions, engagement, and followers. A marketing channel's core question ("did it produce clicks → did they convert?") can only be answered off-platform.

Layer 3 conversion ← quote / topup        [only via PG / quote events]
Layer 2 click      ← go-redirect self-redirect  [not provided by Threads]
Layer 1 engagement ← hourly 6 metrics (views/likes/replies/reposts/quotes/shares)  [Threads API]

The boundary is clear. Threads is primarily responsible only up to "impression → profile." After that, "bio click → landing → conversion" is tracked solely via our own cid (click id). The go.차이사.com/{slug} redirector records the click and assigns a cid, and that value propagates idempotently through 302 → landing cookie → conversion event (/conv).

3.2 Data-first design — four expert lenses + grounding in real measurements

The design was driven not by intuition but by four parallel perspectives (admin UX / ads-conversion / PM-IA / platform ops) → reconciliation. And every decision was grounded in a real snapshot.

Snapshot: 44 posts published (1 account), 13 valid clicks, 126 bot clicks (≈90%), 0 conversions. → Implication: the data is nearly empty. The MVP's first value isn't analytics but making visible "is the pipe alive + can I trust the bot-90% filter + the bio link isn't applied yet." Learning-style analytics (hook×format heatmaps) are deferred until samples accumulate.

The four key decisions that came out of this grounding:

(1) Empty state as a first-class state. With 0 conversions, the screen must not break or mislead. Every widget mandates empty-state copy + a sample-size n badge (n<5 yellow, n=0 gray), with the bot ratio always shown. Four empty states are explicitly classified (collecting / under-sampled / empty bucket / not-connected).

(2) Diagnostic separation of "0 conversions ≠ a bug." I live-verified the whole path — /conv ingestion, token, 302 cid propagation, landing cookie — to confirm the pipe is healthy. Zero POSTs simply means there are no real payments/quotes yet, so the conversion view starts in an "awaiting collection" empty state. The design goal: keep the operator from mistaking an empty screen for an outage.

(3) Blocking double-counting at the schema. Summing partner top-ups (topup, prepayment) and customer quotes (quote, realized revenue) inflates revenue. So I deliberately removed a "total revenue" column from the data contract. Realized revenue=SUM(quote.value) and top-up inflow=SUM(topup.value) are exposed only as two physically separated rows, with the sum slot left empty in the layout to block addition at the source. ROAS numerator = quote only.

(4) The bot filter as the default denominator. At ≈90% bots, click/conversion metrics default to is_bot=0 (+ ip_hash×slug 24h dedup). An unattributed-rate badge sits next to CVR so denominator trustworthiness is always visible.

3.3 Single data contract (VIEW)

Direct access to source tables is forbidden; everything is a VIEW + compute-at-query to force a single interface. The data is small, so materialization (aggregate tables) is deferred (switch on reaching a threshold). The two-DB join (clicks.db ⨝ editorial.db) uses ATTACH.

v_funnel_daily (account × audience × day):
  clicks(is_bot=0) | quote_cnt, quote_value | topup_cnt, topup_value | follower_ctr
  ※ a 'total_revenue' column is intentionally absent → blocks summation at the source
age-normalization: ROW_NUMBER() OVER(PARTITION BY short_id ORDER BY ABS(age_hours-24))
                   — no raw cumulative sort (avoids unfair comparison across post ages)

3.4 Stack right-sizing

Two load-bearing gotchas: ① bind-mounting editorial.db as :ro makes WAL reads throw disk I/O error → fix with an rw mount + read-only at the app connection only. ② go-redirect's @app.get("/{slug}") catch-all swallows all paths → /admin/* must be declared above it.

MVP value proposition: the overview's three widgets — funnel waterfall (views→clicks→conversions) + bot ratio + freshness — convey daily "pipe alive, 90% bots, apply the bio link." Not flashiness; the single signal to watch every day, first.

4. Operational status — scope and limits (in good faith)

This is a record of a well-designed system, not a system with proven marketing outcomes. It's right to state the two separately.

• Current phase = measurement-infrastructure-first. It's a cold start: 44 posts, <100 followers, 0 conversions. Quantitative business outcomes (ER lift, CVR, ROAS) require accumulated samples and real conversions, which is exactly why §3 deliberately leaves learning analytics and the conversion view "awaiting collection." Not claiming outcomes is the honest stance at this point.

• There is, however, qualitative evidence of direction. Content-model changes were driven by operational feedback, not intuition: "bot comments are too tidy / unnatural" → mimic real comments + N<3 DROP; "AI-generated images cause backlash" → real photos only. That is, I iterated toward reducing the bot signature, and recorded the basis for those judgments (off-topic / length-variance of real comments) as data.

• What's proven vs not: Proven — pipeline liveness (generate → queue → publish → 3-layer collection end-to-end), measurement integrity (double-counting / bot filter enforced at the schema), inbound real clicks starting after the bio link. Unproven — whether the content actually drives higher engagement/conversion (causation). That's the next phase's task.

5. Retrospective

• A content model is a refactoring target. enum conflation (TP1) → orthogonal decomposition (2) → filling a missing abstraction (4) is isomorphic to code refactoring. Tracing "why the same failure mode every time" reveals the missing axis.

• A model change entails data migration. The ledger validation logic was a hidden coupling point of the learning loop (TP2). When changing schema/vocabulary, design past-data normalization alongside it.

• Judge build-vs-buy by coupling. The decisive reason for queue+cron over publishing SaaS was coupling, not cost (reuse existing Discord/DB/cron, simple via an idempotent worker).

• Measurement integrity lives in the schema, not the UI. Double-counting is blocked by "remove the total-revenue column," bot contamination by "default denominator is_bot=0" — i.e. enforcing it in the data contract prevents screen-level mistakes at the source.

• Next: close the self-improvement loop with hook×format learning heatmaps once samples suffice; refine attribution quality (cid capture rate / unattributed rate) once real conversions appear.

The one line that runs through both parts: in bot content, "the smell of automatic generation" is a trust cost. Part 1 was the anti-bot threat model; Part 2 was content authenticity and measurement integrity. In the end both are the same question — in "a human picks vetted material, the bot transforms it, the system automates publishing and measurement," where do you draw the boundary?

👈 Part 1: "Designing a Crawler That Beats Three Tiers of Anti-Bot — From TLS Fingerprints to CDP Detection"

A build log of a Threads marketing agent (Hermes) in actual operation. Feedback welcome.

An engineering log of building a Threads marketing agent, split into two parts.

• Part 1 (this post): The data-acquisition layer — a crawler designed around an anti-bot threat model

• Part 2: The generation pipeline — iterative content-model design, scheduled publishing, and performance attribution

TL;DR

In an agent that turns Korean car-community posts into Threads content, the real bottleneck wasn't LLM generation — it was data acquisition (crawling). Each target site enforces a different level of anti-bot protection, so no single collection strategy covers them all.

The key insight was to tier the collection strategy by which signal the anti-bot inspects.

Regardless of how data is fetched, the output is unified into a single normalized schema, and the caller is shielded from per-site complexity. Below, each tier is laid out as hypothesis → experiment → conclusion.

1. Problem definition — the bottleneck is acquisition, not generation

The popular image of "AI marketing automation" puts the weight on generation (prompt → text). In practice, generation converges fast. What eats your time is securing the input: what should we write about? The raw material lives on third-party communities, and those sites block bots.

Two design principles, both learned in operation, governed this whole layer.

Principle 1 — an extraction API ≠ a structure crawler. Early on I used a body-extraction API (e.g. Tavily). For one Bobaedream post: raw HTML 95KB → cleaned output 5KB. Clean, but the tags, post date, and comments (the AJAX payload) were all gone. Our pipeline treats comments as a first-class input — they're central to generation quality — so an extraction API was structurally unfit.

The first fork in tooling, then, is "do I need only body text, or also structure / dynamic content?" The "tidiness" of a cleaning API is information loss.

Principle 2 — bind the scope of automation to the scale of operation. The first design auto-discovered material ("hot tags + last N days + multi-candidate queue"). But scraping unvetted posts by volume only increases topic dispersion. Selecting posts whose engagement is already proven is something human judgment does better than a bot's heuristic scoring. So I fixed the premise to "a human picks the source URL," and let the bot focus on extracting structure losslessly from that one URL.

That made the collection engine's single responsibility crisp: "one human-chosen URL → losslessly normalize {title, body, comments, images}." It sounds trivial, but per-site HTML structure and anti-bot all differ, so a single implementation is impossible.

2. Interface design — encapsulating per-site complexity

Even when the strategy diverges per site, the caller must not know about that branching. A single helper (fetch_source.py) takes only a URL, dispatches internally, and returns one schema.

python3 bin/fetch_source.py "<url>"   # site-agnostic → normalized JSON

detect_site(url)   # domain → {bobaedream, clien, dcinside, fmkorea}
derive_board(url)  # derive board code / post id from the URL (never hard-code)
→ single schema: {ok, site, title, body, images[], comments[{nick,text}], recommend_count, comment_count}

Design decision — board identifiers are always derived from the URL. DCInside gallery ids, Clien board names, and Bobaedream code/No are all encoded in the URL. Baking them in as constants means a code change for every new board. Regex derivation supports any board on the same site with zero changes — in effect, the open–closed principle applied to the data layer.

Every per-site hell that follows is locked behind this abstraction. The caller's contract is simply "URL in, normalized JSON out."

3. Tier 1 — direct urllib requests (Clien, DCInside, Bobaedream)

What these three share: body and comments are server-rendered HTML, and the anti-bot is only at the IP-reputation level. So a direct urllib request (with a CookieJar) suffices. The only variables are the comment-collection path and the selectors.

DCInside is the trickiest. The e_s_n_o token is a one-time value refreshed per view page, so it can't be cached — it forces a two-step dance: "extract token → POST for comments with that token." Bobaedream is scraped via the mobile view with a mobile UA, but "best" posts need special handling: re-parse the real code/No from the inline markup to build the comment endpoint.

Tier 1 conclusion: bodies are all server-rendered (no JS execution needed) and there's effectively no anti-bot. The only variable is the comment path. But the fourth site breaks that premise.

4. Tier 2 — FMKorea: IP and TLS fingerprint, a double gate ⭐

FMKorea's body is server-rendered too. The variable is the anti-bot. I narrowed the blocking surface with hypothesis-driven experiments.

4.1 Experiment log

hypothesis / attempt                          result
──────────────────────────────────────────────────────
H1: datacenter IP is the problem
  VPS + curl                              →  HTTP 430 (immediate)
H2: residential IP fixes it
  residential IP + curl, cold first call  →  200 (passes!)
  residential IP + curl, subsequent calls →  430 "FMKorea security system"
H3: header spoofing gets around it
  residential IP + curl + full browser headers + h2 → 430 (no change)
H4: a real browser passes
  residential IP + real headless Chrome   →  200, 112KB, full body + comments

H2's cold-pass was a trap. It tempted a hasty generalization — "residential IP means curl can automate this" — but on replay that was a lucky one-off and it converged to 430. The blocking surface was two layers: ① IP reputation (datacenter), ② TLS fingerprint (curl). The decisive clue was H3: perfect header spoofing still failed, meaning the anti-bot inspects the TLS layer beneath, not the L7 headers.

4.2 Why header spoofing is futile — the TLS fingerprint (JA3/JA4)

An https connection sends a ClientHello during the TLS handshake, before any real traffic. That ClientHello carries the cipher suites, extensions, elliptic curves, and signature algorithms the client supports — and their order. The key fact: this list and ordering are unique per implementation. curl, Chrome, and Safari each differ. Hashing it gives the JA3/JA4 fingerprint.

The User-Agent is a self-reported application-layer value, freely forgeable. The TLS fingerprint, however, comes from the TLS-stack implementation itself. So even if you spoof the header to Safari, the handshake fingerprint is still curl. The anti-bot flags this mismatch ("UA says Safari but JA3 says curl") as a lying bot and returns 430. Add a behavioral signal (high frequency from one IP) and you get the immediate block right after the cold pass.

In short, L7 spoofing can't fake an L6 fingerprint. Passing requires satisfying both "residential IP (reputation)" and "real browser TLS stack (fingerprint)" at once.

4.3 Solution architecture — delegate to a residential node via Tailscale

[VPS / hermes-agent]  (datacenter IP)
   ├─ Bobaedream/Clien/DCInside → direct request (Tier 1)
   └─ fmkorea → delegated call (Tailscale tailnet)
                  │
                  ▼
        [Mac mini fetch service]  (residential IP + real Chrome)
                  │  chrome --headless=new --dump-dom <url>
                  ▼
              fmkorea.com  → passes IP + TLS fingerprint → 200

An always-on Mac mini runs real Chrome headless to clear both gates at once; the VPS delegates to this node only for FMKorea over Tailscale and just receives the result JSON.

Rejected alternatives: ① Routing all VPS traffic through a Tailscale exit node → single point of failure, complex policy routing, latency on all traffic. Selective delegation for FMKorea only is far lower-coupling. ② curl-impersonate (a build that mimics Chrome's JA3) → works, but with a real Chrome available the operational simplicity wins. ③ An extraction API (Tavily) → its servers do the fetching, so IP evasion is moot (you only get a shell).

4.4 Trade-off — Chrome CLI vs Playwright

First, a common myth: Playwright is not what beats the anti-bot. What passes the TLS fingerprint is "real Chrome," not the driver. Playwright is just a question of how you drive that Chrome.

FMKorea posts are server-rendered with title/body/comments all in the initial DOM, so no interaction is needed. A single --dump-dom returns 112KB including comments. So I chose the direct CLI: zero dependencies, simple code, minimal detection surface. If dynamic loading (load-more / infinite scroll / login) ever appears, escalating to Playwright then is the rational move. YAGNI.

Implementation detail (non-blocking termination): a fresh-profile Chrome doesn't exit immediately after --dump-dom (background tasks), causing a ~45s timeout per request. The fix: stream stdout and kill the process the moment </html> appears, clean the profile lock per call, and serialize with a global lock. Response dropped to 3–5s.

Tier 2's premise: "we can launch a real browser ourselves." Tier 3 breaks that premise.

5. Tier 3 — Upwork: the automation browser itself is the detection target ⭐

A separate task required collecting Upwork job data (rate/proposal distribution per keyword = a bidding price dataset). Every Tier-2 weapon was neutralized.

ℹ️ Upwork collection is a separate task, unrelated to this marketing agent's material pipeline. But it presented a fundamentally different class of anti-bot (detecting automation itself) and therefore demanded a completely different crawling paradigm (userscripts) — so I record it here as the final tier of the threat model.

5.1 Failure matrix

attempt                                               result
──────────────────────────────────────────────────────────
curl / requests                                    →  403
public RSS                                         →  410 Gone (deprecated)
Playwright (bundled Chromium)                       →  Cloudflare infinite challenge
  + stealth plugin                                 →  infinite challenge
  + system Chrome (channel:chrome), headed         →  infinite challenge
  + real login profile + cf_clearance cookie       →  infinite challenge

The last line is the crux. Even with real Chrome + headed + a valid login session + a passing cookie (cf_clearance) all present, Cloudflare kept re-challenging. Failure despite satisfying every condition that beat Tier 2 → a signal that the blocking criterion isn't "identity" but something else.

5.2 Probable cause — Cloudflare's automation (agency) detection

The probable cause is runtime detection of automation-control signals, chiefly CDP (Chrome DevTools Protocol) exposure. CDP is the protocol Playwright/Puppeteer/Selenium use to drive a browser externally. In other words, an open CDP session is itself the signal that "this browser is being controlled by code."

Enterprise Cloudflare evaluates TLS, behavior, and automation markers together. Yet H4 (real Chrome + headed + login + cf_clearance) satisfied every identity/session signal and still got blocked. That narrows the remaining variable to one: the agency signal — "is this being controlled by automation?"

⚠️ Limits of proof (in good faith): I did not run the experiment that isolates CDP as the sole cause (e.g. a CDP-free build as control). Cloudflare uses multiple signals, so other automation markers like navigator.webdriver may also have contributed. What I can assert is only this: even satisfying every identity/session signal, an automation-controlled session does not pass.

The depth of the threat signal is one level different — that's the core.

• TLS fingerprint (Tier 2): "who (which implementation) connected" — identity

• Automation detection (Tier 3): "is this session controlled by code" — agency

cf_clearance, real Chrome, and login all satisfy the identity signal, but the agency signal can't be turned off as long as you use an automation tool. → Practical conclusion: you cannot reliably collect Upwork with Playwright/Selenium-class tools. It's not about evasion difficulty; the fact of automation is the detection surface.

5.3 Paradigm shift — a userscript inside a human session

If what's detected is "a CDP-controlled browser," then run the code inside a browser with no CDP — a session the user opened themselves. That's a Tampermonkey userscript: ordinary JS running in the page context, with no CDP connection and no navigator.webdriver. From Cloudflare's view there's no automation marker to detect.

// ==UserScript==
// @name         Upwork Job Harvester → CSV
// @match        https://www.upwork.com/nx/search/jobs/*
// @grant        GM_getValue
// @grant        GM_setValue
// ==/UserScript==
const TARGET_PER_KEYWORD = 200;  // auto-stop once the per-keyword target is hit
const MIN_DELAY = 4500, MAX_DELAY = 8000;  // randomized delay between page turns — avoid behavioral signals

Implementation points:

• State persistence — accumulate data in GM_setValue so nothing is lost across page transitions (SPA routing). Auto-paging + dedup by job id (~0...).

• Behavioral-signal management — even a real browser gets flagged on high frequency, so 4.5–8s randomized delays.

• Tiered data sources — progressively enrich fields from list cards / the search page's window.__NUXT__ / the detail page's __NUXT_DATA__ (Nuxt3 flat array).

5.4 Decision framework — threat signal → strategy

Generalizing all three tiers into one decision tree:

The core judgment: if the anti-bot inspects only identity, we drive a browser to get through. If it inspects agency (whether it's automated), then the very fact that we drive the browser is grounds for blocking, so the code must be carried inside a human session. The latter trades away full automation to eliminate detection at the root. Designing well is recognizing and choosing that trade-off explicitly.

6. Data-trust gate — collected ≠ usable

A successful fetch isn't usable yet. A quality gate sits before generation.

Comments N<3 → DROP (no synthetic fallback). This is the most important rule. A bot's formulaic "empathy / rebuttal / verification" comment set is too tidy, and that tidiness itself becomes a bot signature. Real community comments are different: an off-topic top reply ("…bought 10 Samsung shares yesterday"), high length variance ("Fighting!!!"), chit-chat chains.

So I use the actually-fetched comments as the source and mimic their tone / length / off-topic distribution. Verbatim copying is banned (author protection). If fewer than 3 sources exist, the post is dropped rather than padded with synthetic fillers.

A synthetic comment's naturalness comes not from "being well-written" but from "resembling the real thing." The closer you get to an ideal debate structure, the more it reeks of a bot — the goal is to mimic the flaws, not the polish.

Also: a hard-reject filter (death/serious injury, legal disputes, politics, hate), and images are original real photos only (no AI generation — the "🎨 AI image" label is itself a trust cost for a bot).

7. Retrospective

• Threat-model-first design paid off. Classifying by "what the anti-bot inspects" rather than "how do I scrape" means a new site drops cleanly into a tier via the decision tree.

• The value of hypothesis-driven debugging. I once drew a wrong conclusion (curl can automate this), fooled by H2's cold pass. A replay experiment isolated the real cause — the TLS fingerprint. The discipline of not trusting a one-off success is the key.

• The abstraction boundary. Locking per-site branching and anti-bot evasion behind fetch_source lets the generation pipeline (Part 2) consume one schema, blind to the data's origin.

• Known limits (honestly): the Mac mini node is a single point of failure and can't fetch while asleep; Chrome runs serially (3–5s/request), a bottleneck for bulk collection → multi-node / browser-pool is the next task. --dump-dom captures only the initial DOM, so exhaustive comment-pagination would require escalating to Playwright.

State reached: four sites (Bobaedream, Clien, DCInside, FMKorea) collected through a single interface; FMKorea cut from ~45s to 3–5s via the delegate node; Upwork given a collection path through userscripts in an environment where automation is structurally impossible. Every source unifies into one normalized schema.

Part 2 covers turning this normalized material into community-style content without a bot signature (the iterative generation-skill design), plus scheduled publishing and performance attribution.

A build log of a Threads marketing agent (Hermes) in actual operation. Feedback welcome.

"Make it faster" sounds like a vague mandate. It isn't. Performance work turns out to be one of the most learnable parts of backend engineering, because it almost always comes down to the same three moves. These are the three I shipped in production — and they took our QPS up more than 10x.

Let me walk through them the way I actually think about them now: why the bottleneck exists → how to fix it → how much it actually moved the needle.

First, what is QPS — and why does it hit a wall?

QPS (Queries Per Second) is just how many requests you handle per second. But you can't will it higher. It's bound by a formula:

QPS ≈ number of concurrent workers / time to process one request

The checkout counter analogy makes this click. Think of a worker (a thread or a connection) as a cashier, and request-processing-time as how long it takes to ring up one customer.

• 1 cashier, 0.5s per customer → 2 customers/sec = 2 QPS

• Speed them up to 0.1s per customer → 1 cashier now does 10 QPS (you shrank the denominator)

• Add 2 more cashiers → at 0.1s each, 30 QPS (you grew the numerator)

QPS = cashiers / time-per-customer   →   3 / 0.1s = 30 QPS

The trouble starts when customers pour in faster than that. Hit a 30-QPS shop with 40 customers per second, and the line grows by 10 every second → wait times explode → customers walk out (timeouts) → the shop seizes up (an outage). Servers behave exactly the same way.

So there are only two ways to raise QPS:

1. Shrink the denominator (time per request) → caching, DB optimization

2. Pull work out of the denominator — anything that doesn't have to happen inside the request → async / batching

The three strategies below attack exactly these two levers. And in practice, whenever you start tuning, you reach for them in this order — it's the order of best bang-for-buck: Caching → DB optimization → Async/Batching.

1. Caching — "Never compute the same thing twice"

The idea

Caching is fundamentally about storing the result of an expensive operation in a cheap place and reusing it for a little while. The two words that matter are expensive and a little while.

How expensive? The response-time gap is why caching is the #1 bang-for-buck optimization:

So if a result took 50ms to fetch from the DB once, caching it in Redis means you can return it in ~1ms from then on. The per-request time (the denominator) drops by 50x — and QPS jumps right along with it.

Multi-layer caching

In production you rarely use just one cache. You stack layers, and the closer to the user you can stop the request, the better. In the order a request travels through them:

User request
   │
   ▼
① CDN cache (CloudFront/CloudFlare)   ← if it hits here, it never reaches your server (best)
   │ miss
   ▼
② App memory cache (in-process LRU)   ← 0.001ms, but lives per-server
   │ miss
   ▼
③ Redis (distributed cache)           ← 1ms, shared across all servers
   │ miss
   ▼
④ Database (the source of truth)      ← 50ms; reaching here is a "cache miss"

• ① CDN — static assets (images, JS, CSS) or public API responses that don't change. It ends at an edge near the user, so your servers don't even see the traffic.

• ② App memory — fastest, but independent per server: 10 servers means 10 copies and painful invalidation. Best for tiny, rarely-changing data (exchange-rate tables, shared codes).

• ③ Redis — shared by all servers, so invalidation is one shot. The workhorse of production caching.

Heads-up on naming: some texts call these L1 (app memory) / L2 (Redis) / L3 (DB or CDN). There, the numbers go by closeness to the data/CPU (L1 is closest to the app), which is the opposite direction from the request-flow numbering above (①CDN first). It's easy to mix up, so I numbered purely by request order here.

The genuinely hard part: what to cache and for how long

Writing cache code is easy. The hard part is the TTL and invalidation strategy. There's a reason for the old joke that the two hardest problems in computer science are cache invalidation and naming things.

Strategy depends on the nature of the data:

Invalidate-on-write pattern:

// Read: cache first, fall back to DB, then populate the cache (cache-aside)
async function getProduct(id: string) {
  const cached = await redis.get(`product:${id}`);
  if (cached) return JSON.parse(cached);          // cache hit → 1ms

  const product = await db.product.findUnique({ where: { id } }); // miss → 50ms
  await redis.set(`product:${id}`, JSON.stringify(product), "EX", 300); // 5-min TTL
  return product;
}

// Write: update the DB, then DELETE the cache (next read repopulates it)
async function updateProduct(id: string, data) {
  const product = await db.product.update({ where: { id }, data });
  await redis.del(`product:${id}`);   // ← invalidation. This one line is the point
  return product;
}

⚠️ Common mistake: after a write, deleting the cache (del) is usually safer than updating it (set). If two writes update concurrently, a stale value can win the race; deleting means "fetch fresh from the DB next time," which always reflects the latest state.

💡 What I shipped — the "view-count batching" on 3w CMS

A post's view count is the textbook "read-heavy, accuracy-doesn't-really-matter" kind of data. But the original design fired a DB UPDATE on every single view.

Before — 1 view = 1 write query:

-- runs every time someone opens a post
UPDATE posts SET view_count = view_count + 1 WHERE id = ?;

The problem: a popular post getting 500 readers/sec means 500 write queries per second hammering the same row. Concurrent UPDATEs on one row create row-lock contention, so bumping a view counter ends up slowing the entire DB. Writes can't be cached and can't be offloaded to replicas — it's the most painful kind of load.

After — Redis counter + a 5-minute batch flush:

// On view: increment a Redis counter, not the DB (0.5ms, no lock)
await redis.incr(`view:post:${postId}`);

// A job every 5 minutes: collect and apply to the DB in one shot
async function flushViewCounts() {
  const keys = await redis.keys("view:post:*");
  for (const key of keys) {
    const postId = key.split(":")[2];
    const count = await redis.getdel(key);  // read value and delete atomically
    await db.post.update({
      where: { id: postId },
      data: { viewCount: { increment: Number(count) } },
    });
  }
}

Result: 1,500 views over 5 minutes hit the DB as a single UPDATE. Write load dropped by 1,500x. The count can lag by up to 5 minutes, but for view counts that delay is completely harmless — the trade-off is a perfect fit.

This is caching (temporary storage in Redis) fused with batching (collect, then flush once) — which leads straight into strategy #3.

2. Database optimization — where 80% of bottlenecks live

Application code usually runs in microseconds, but the DB operates in milliseconds to hundreds of milliseconds — so most backend bottlenecks happen at the DB. That's why it's the next thing you touch after caching. Three things come as a set.

The big premise: the fastest query is the one you never ran. The #1 priority in read optimization isn't writing a clever query — it's reducing how often you reach the DB at all. So in practice, before hitting the DB directly, you filter reads in this order:

1. Can the cache answer it? — the multi-layer cache from #1 stops a large share of reads before the DB. Hot data barely reaches it.

2. Can several queries collapse into one? — fold N+1 into an IN-query / JOIN / DataLoader (see batching in #3). 11 round-trips → 2.

3. Are you fetching only what you need? — specific columns instead of SELECT *, paginate to cap rows, drop needless JOINs.

So the index/pool/replica below are there to serve the queries that still have to reach the DB after all that filtering — quickly and reliably. "Run fewer queries" always comes before "run queries faster."

2-1. Index optimization — "the index at the back of a book"

Without an index, the DB has to read the table from start to finish (a Full Table Scan) to find the rows you want. A million rows means scanning all million. An index is like the index pages at the back of a book — a pre-sorted structure (a B-Tree) that says "this value is on page N."

A table and an index are not the same thing. The table is the original data (the book's body); the index is an auxiliary lookup to find it fast (the back-of-book index pages).

• An index isn't new data — it's a derived copy of select columns, sorted. Drop it and no data is lost; only search gets slow again → so you add/drop them freely, even in production.

• When you insert new data, the DB automatically updates the table and its indexes together. ↔ This is exactly why "too many indexes slow down writes."

• One table can have several indexes for different search patterns, and the DB picks the most useful one per query. (This is why the same SQL can be fast or slow depending only on which indexes exist.)

Concrete example — fetching one user's orders from a 100k-row table:

SELECT * FROM orders WHERE user_id = 'u_123' ORDER BY created_at DESC LIMIT 20;

• No index: read all 100k → filter by user_id → sort. ~120ms.

• Index on user_id only: finds that user's orders (say 500) fast, but still sorts in memory. ~15ms.

• Composite index (user_id, created_at DESC): find and sort finish inside the index at once. ~0.5ms. 👈

Column order in a composite index matters. (user_id, created_at) is ideal for "narrow by user_id, then sort by created_at," but it can't serve a query that searches by created_at alone. The rule of thumb: equality (=) columns first, range/sort columns last.

⚠️ Trade-off: indexes aren't free. Every INSERT/UPDATE has to update the indexes too, so writes get slower and you use more storage. Slapping an index on every column "just in case" backfires. Look at your actual query patterns and add only what you need.

2-2. Connection pooling — "don't open a new connection every time"

Opening a new DB connection costs tens of milliseconds (a TCP handshake + authentication). Doing that per request is absurd when the query itself is 1ms — you'd burn 30ms on the connection alone.

A connection pool pre-creates N connections and lends them out and takes them back for reuse.

[req A] ─┐                  ┌─ [conn 1] ─┐
[req B] ─┤  Pool (reuse)    ├─ [conn 2] ─┤── DB
[req C] ─┘                  └─ [conn 3] ─┘
         (wait queue)

Sizing the pool — too small and too big both hurt:

• Too small: requests queue up waiting to borrow a connection → latency.

• Too big: the DB sees a flood of concurrent connections → its memory/CPU blows up, since each connection costs the DB memory.

• Rule of thumb: start around CPU cores × 2–4 and tune with load tests.

As you add app servers, each holds its own pool, so total DB connections multiply — which is why you often put a pooler like PgBouncer in front as an extra layer.

2-3. Read replicas (read/write splitting)

Most services have a read:write ratio of 9:1 or higher (write one post, hundreds read it). So spreading reads across several replicas takes a huge load off the single master.

                ┌─ writes (INSERT/UPDATE/DELETE) ─→ [Master DB]
[application] ──┤                                       │ replication
                └─ reads (SELECT) ─→ [Replica 1] [Replica 2] [Replica 3]

Effect: with reads split across 3 replicas, the master focuses on writes, and read capacity effectively triples.

⚠️ Replication lag: data written to the master takes anywhere from a few to a few hundred milliseconds to copy to the replicas. So a read-after-write case (e.g., post a comment, then immediately view the comment list) will read from a replica and show "wait, where's my comment?" Those reads must be forced to the master.

// Normal reads go to a replica
const posts = await readClient.post.findMany(...);

// Reads that must reflect a just-made write go to the master
await writeClient.comment.create({ data });
const comments = await writeClient.comment.findMany(...); // read from master

How the three connect

These aren't separate — they're one bundle: indexes make a single query fast, the pool removes connection waste, and replicas spread out the reads. Cut a query from 100ms to 1ms with an index, and the same pool can now serve 100x more requests — so the three effects multiply.

3. Async + batching — "pull everything non-essential out of the request"

If #1 and #2 were about making each task faster, #3 is the mental flip: take the task out of the user's critical path entirely.

3-1. Async processing via a message queue

Think about signup. From the user's perspective, "signup complete" means the user exists in the DB — that's it. But if you send the welcome email (an external SMTP call, 500ms) synchronously inside that request, the user just sits there waiting for an email to send.

Before — synchronous (signup API takes 510ms):

async function signup(data) {
  const user = await db.user.create({ data });   //  10ms
  await sendWelcomeEmail(user.email);            // 500ms ← the user waits on this
  return user;                                    // 510ms total
}

After — enqueue and respond immediately (signup API takes 12ms):

async function signup(data) {
  const user = await db.user.create({ data });   // 10ms
  await queue.enqueue("send-welcome-email", {     //  2ms (just enqueue)
    email: user.email,
  });
  return user;                                    // 12ms total ← user response
}
// A separate worker consumes the queue and actually sends the email (in the background)

Result: 510ms → 12ms. Per-request time dropped ~40x, and QPS rises with it. The email arrives a second later and nobody notices.

Good candidates to push onto a queue:

• Email/SMS, push notifications

• Image resizing / thumbnail generation

• Stats / analytics aggregation

• Third-party API calls (slow, failure-prone)

Tools: Kafka, RabbitMQ, Redis Streams, AWS SQS, etc. As a bonus you get retries and failure isolation for free — if the email server is briefly down, jobs pile up in the queue and send once it recovers, while signup itself never blocks.

3-2. Batching — "collect many, do them at once"

Every network/DB round-trip has a fixed cost. Sending 100 items one-by-one (100 trips) is far more expensive than bundling them into a single trip.

Classic example — sending analytics events:

// Before: send on every event → 1000 round-trips
track(event) { await http.post("/analytics", event); }

// After: buffer, then send once per 100 events or every 1s → 10 round-trips
const buffer = [];
track(event) {
  buffer.push(event);
  if (buffer.length >= 100) flush();
}
function flush() {
  http.post("/analytics/bulk", buffer.splice(0));  // everything collected, in one shot
}
setInterval(flush, 1000); // flush every 1s even if 100 isn't reached

Fixing N+1 queries is batching too (the DataLoader pattern):

// Before (N+1): 10 posts → 10 separate author lookups
for (const post of posts) {
  post.author = await db.user.findUnique({ where: { id: post.authorId } });
}
// 1 (posts) + 10 (authors) = 11 queries

// After (batched): collect author ids into one IN-query
const authorIds = posts.map(p => p.authorId);
const authors = await db.user.findMany({ where: { id: { in: authorIds } } });
// 1 + 1 = 2 queries  (DataLoader does this collecting automatically)

💡 What I shipped — the "50ms-window notification aggregation" on 3w CMS

In our notification system, a single user sometimes got a burst of events in a tiny window (e.g., 10 messages in a group chat within one second). Firing a push per event meant:

• The user's phone buzzed 10 times → terrible UX

• 10 outbound calls to the push provider (FCM/APNs) → load + cost

Fix — bundle within a 50ms window: when a notification arrives, don't fire immediately. Wait 50ms; if more notifications for the same user arrive, merge them and send one "10 new messages" push.

// Open a 50ms window per user and collect notifications that arrive within it
function onNotification(userId, payload) {
  if (!windows.has(userId)) {
    windows.set(userId, { items: [] });
    setTimeout(() => flushWindow(userId), 50);  // send once, 50ms later
  }
  windows.get(userId).items.push(payload);
}

function flushWindow(userId) {
  const { items } = windows.get(userId);
  windows.delete(userId);
  // one item → send as-is; many → merge into "N new notifications" and send once
  sendPush(userId, summarize(items));
}

Result: during bursts, push calls dropped to a fraction of before, and users got one clean summary instead of a notification bomb. Performance and UX, fixed in one move. 50ms is imperceptible to a human, so the trade-off was practically free.

3-3. Debouncing / throttling — batching on the client side

Rapidly repeating requests, like search autocomplete, should be bundled or thinned out.

• Debounce: "fire one request only after 300ms of no further input." Type S, Se, Seoul within half a second → not 3 requests, just 1 for Seoul.

• Throttle: "at most once per second, no matter how often it fires." Thins out things like scroll events that fire hundreds of times a second.

Blocking on the client is the cheapest optimization of all, because the request never reaches the server.

Wrapping up — why this order, and why the effects multiply

There's plenty beyond these three — horizontal scaling (load balancer + multiple servers), HTTP/2, gzip/brotli compression, separating static assets — but applying just these three properly usually lifts QPS by 10x or more. So when you start tuning, you almost always go in this order (best bang-for-buck first):

The key is that the three multiply. Caching keeps 80% of reads from ever reaching the DB (5x), indexes speed up the remaining 20% (10x), and heavy work moves to a queue — the gains don't add, they compound, and total QPS jumps by tens of times.

And the thread running through all three: consciously choosing your trade-offs. View counts can lag 5 minutes, notifications can bundle for 50ms, emails can arrive a second late. Knowing what's allowed to be slightly wrong or slightly late — that's the real skill in performance engineering.

Thanks for reading! If you've shipped a performance fix that fit one of these buckets — or broke one of these rules and paid for it — I'd love to hear about it in the comments.

42.8% HTTP error rate, 9 minutes into a load test. The culprit was a single updateMany line carelessly written during AI-assisted coding.

1. Discovery — "Why are we doing 9.5 TPS when the target is 75?"

We ran k6 against a load-test environment, targeting 75 TPS of chat traffic. The numbers came back ugly.

The first suspect was always going to be connection pool exhaustion. But Postgres was only using 11 of 27 connections, and Centrifugo publish was 100% successful. Every signal pointed at the app server's database queries themselves.

2. The Real Culprit — 40P01 deadlock detected

docker logs witim-web showed all the 5xx lines in identical shape:

prisma:error
Error occurred during query execution:
ConnectorError(... QueryError(PostgresError {
  code: "40P01", message: "deadlock detected", severity: "ERROR",
  detail: Some("Process 111892 waits for ShareLock on transaction 158028;
                blocked by process 111881.
                Process 111881 waits for ShareLock on transaction 158027;
                blocked by process 111892.")
}))

Not P2024 (pool timeout). Not ETIMEDOUT. Pure PostgreSQL row-lock deadlock. Sixteen concurrent transactions were locking the same set of rows in different orders and crashing into each other.

3. Code Audit — Wait, We Do 4–5 Writes Per Message?

The offending handler, POST /api/channels/[id]/messages, was doing four to five serial writes with no transaction wrapper:

#4 is the killer. When users A and B send messages to the same channel concurrently:

Tx-A: UPDATE ChannelMember WHERE channelId=X AND userId != A
        → locks rows B, C, D, ... one at a time, in scan order
Tx-B: UPDATE ChannelMember WHERE channelId=X AND userId != B
        → locks rows A, C, D, ... one at a time, in scan order

updateMany is one SQL statement, but with no ORDER BY, Postgres picks lock order based on the plan. Cache state, parallel worker distribution, and table statistics all influence it — meaning the order can change between two concurrent executions of the same SQL.

Direct Evidence From the PG Log

Every deadlock had a different CONTEXT line:

(page, slot) is different every time → lock order is non-deterministic. Hypothesis confirmed.

4. Honest Confession — This Was an Embarrassing AI-Coding Mistake

Let me come clean. This updateMany was generated during AI-assisted coding.

// Intent: "When a message is sent, bump unreadCount by 1 for every other member"
await prisma.channelMember.updateMany({
  where: { channelId, userId: { not: senderId } },
  data: { unreadCount: { increment: 1 } },
});

Syntactically perfect. Works flawlessly in a single-channel, single-message, single-user test. PR review passed.

But it silently skipped two questions that should have been asked:

1. "Should we really write to N rows on every single message?" — A channel with 200 members means 200 row writes per message. 100 users sending = 20,000 writes/sec. At that volume, RDBMS row writes per message are structurally inefficient.

2. "What happens when two people post to the same channel at the same time?" — That's the core usage pattern of any messaging app. The AI didn't think about it. And neither did I when I merged the PR.

This is the AI-coding trap. Code that looks like it works lets you skip thinking about concurrency scenarios. Without the load test, this would have detonated in production the same way.

5. Prescriptions on the Table — How Far Should We Go?

The issue listed three tiers of fixes:

P0 (immediate):

• Deadlock retry interceptor for 40P01

• Rewrite updateMany as raw SQL with ORDER BY id FOR UPDATE → deterministic lock order

P1 (short term):

• Move unreadCount to Redis HINCRBY, periodically flushed to DB

P2 (medium term):

• Partition Message INSERTs by channelId hash

P0 alone makes the deadlock go away. But 1 message = N row writes stays. At 100K CCU, this will explode again — just in a different form.

P1 (Redis) is faster but introduces Redis ↔ Postgres synchronization complexity. unreadCount temporarily diverges from Postgres, and a failed Redis flush can permanently skew the count.

So we chose a different direction.

6. What We Picked — The lastReadAt Column

Core idea:

"Don't write a counter on every message. Compute the count when you read."

Drop the ChannelMember.unreadCount column. Replace it with a single ChannelMember.lastReadAt timestamp.

Before — write-heavy

[Send 1 message]
  ├─ INSERT Message                              (write 1)
  ├─ UPDATE ChannelMember unreadCount++ for N    (write N) ← deadlock vector
  └─ UPDATE ChannelMember unreadCount=0 (sender) (write 1)

→ 1 message = (N+2) writes

After — read-on-demand

[Send 1 message]
  └─ INSERT Message                              (write 1)
                                                          ← done.

[Fetch channel list]
  └─ SELECT count(*) FROM Message
       WHERE channelId = $1 AND createdAt > lastReadAt   (read, hits index)

[Enter channel / mark as read]
  └─ UPDATE ChannelMember SET lastReadAt = NOW()
       WHERE userId = $1 AND channelId = $2              (write 1, own row only)

Why This Kills the Deadlock

updateMany WHERE userId != sender disappears. Sending a message no longer touches other users' rows. The lock-contention vector is removed from the code entirely.

lastReadAt updates only touch your own row. User A only ever updates A's ChannelMember row, user B only B's. There is no scenario where two transactions compete for the same row — it's eliminated by construction, not by retry logic.

Why This Isn't a Performance Regression

At first glance, "running count(*) every time" sounds slower. It isn't, because:

1. Reads are cache-friendly. With a composite index on (channelId, createdAt), the count becomes an index range scan.

2. Read frequency << write frequency. Users see the channel list when they open the app or enter a channel. Messages are sent dozens to hundreds of times per second. Moving N writes → 1 read is an enormous win on its own.

3. Reads can be served from a read replica. Write locks cannot.

Sequence Diagram (Conceptual)

[Before — count at write time]
User A ─── POST /messages ───▶ Web
                                 ├─ INSERT Message
                                 ├─ UPDATE ChannelMember(N rows) ← 💥 deadlock
                                 └─ 200 OK

[After — count at read time]
User A ─── POST /messages ───▶ Web
                                 ├─ INSERT Message
                                 └─ 200 OK                       ← done

User B ─── GET /channels  ───▶ Web
                                 ├─ SELECT channels ...
                                 ├─ SELECT count(*) FROM Message
                                 │    WHERE createdAt > lastReadAt
                                 └─ [{ id, name, unread: 3 }, ...]

User B ─── POST /channels/X/read ▶ Web
                                 └─ UPDATE ChannelMember
                                      SET lastReadAt = NOW()
                                      WHERE userId = B           ← own row only

7. Migration Strategy — Don't Flip Everything At Once

Phased rollout for backward compatibility:

1. Phase 1: Add the lastReadAt column. Keep the existing unreadCount write path untouched.

2. Phase 2: Switch only the read API to the new approach (count(*) WHERE createdAt > lastReadAt).

3. Phase 3: Re-run the load test. Confirm 75 TPS passes cleanly.

4. Phase 4: Delete the unreadCount write code and drop the column.

By the end of Phase 2, the deadlock vector is already gone. Why? Once the read path is based on lastReadAt, a wrong unreadCount value is no longer visible to users. After that, we can clean up the write path on our own schedule, with no production pressure.

8. Lessons — AI Coding and Concurrency

This incident wasn't a one-line bug. It was a thinking-pattern trap.

• The AI-generated updateMany is a correct answer to "How do I express this in Prisma syntax?"

• It is not an answer to "In a messaging app, how does this SQL acquire locks when two users post to the same channel simultaneously?" Because nobody asked that question.

• The faster AI-generated code lands, the more human review of concurrency, isolation, and scalability matters. That's the actual value of a PR review.

One more lesson. Load testing belongs before production, not after. Without an isolated load-test environment, this would have detonated under 100K CCU traffic. We'd have been chasing a 5xx storm caused by deadlock cycles in production. The environment itself prevented one incident.

I wanted to know exactly how many TPS our messenger's send API could survive — and where it would actually break. So I mirrored production into a 7-node load-test environment, and instead of treating it as a one-shot setup, I designed it to be reproducible on demand.

This post is a record of that process — what we wanted to measure, what had to be prepared first, and how the environment itself was wrapped so anyone can re-run the whole thing.

Why "100 MPS" isn't an answer

People love to say "our service handles 100 MPS." Try answering any of the following with just that number:

• At what WebSocket concurrency was the 100 MPS measured?

• What message type? (Plain text? Mentions? Attachments?)

• Into what kind of channels? (5-member? 50-member? Thousands?)

• For how long did the load hold?

• What did CPU, DB connections, Redis, Centrifugo look like during the run?

• What were the 4xx / 5xx rates and the REST→WS p95/p99 delivery latency?

A single number that strips all of those variables is meaningless. So the goal of this project was not "produce a bigger number" — it was "build a measurement environment that can answer all of the above."

There was one more design rule.

Don't load a single channel with thousands of members. Distribute the load across hundreds of small channels (5-/50-member mix).

Our message API enforces a msg:{userId}:{channelId} rate limit (10 requests / 10 seconds, i.e. ~1 TPS per pair). Pile load onto a single channel and you hit 429 right around 100 TPS, well before any real backend pressure shows up. To exercise 1k TPS under production-like conditions you need at least 1,000 distinct sender/channel pairs, ideally 2,000+. The dataset itself has to be designed around the production traffic shape.

Topology — a faithful copy of production

If the test topology differs from production, the numbers will too. So we provisioned 7 Vultr VMs on the same plans as production, including the separate ALB layer.

k6 / WS Client
    │
    │ HTTP /api/*  +  WS /connection/*
    ▼
HAProxy ALB
    │ :80   public
    │ :8000 internal Centrifugo API
    │
    ├── /api/*, /api/centrifugo/* ──► Next.js App
    │                                      │
    │                                      │ App → HAProxy:8000 publish
    │                                      ▼
    └── /connection/* ───────────────► Centrifugo C1 / C2 (round-robin)
                                              │
                                              ▼
                                       Redis (broker / cache / rate limit)

Next.js App writes → PostgreSQL

Reusable infrastructure — Makefile + Terraform as a single entry point

The biggest trap with a load-test environment is "set it up once, never touch it again." When a 1k TPS smoke breaks the DB and you come back a week later to run a knee sweep, if anyone has to remember the PG password or where cloud-init last stalled, the whole afternoon is gone.

So everything is wrapped into one Makefile-driven flow. Terraform manages all 7 hosts in a single state, and Make sequences up → wait → cutover → smoke → down on top of it.

# standard flow
make up         # provision 7 VMs + cloud-init + auto-refresh HOSTS.env
make wait       # wait for cloud-init on all 7
make secrets-check   # verify compose/.env (HMAC, API key) consistency
make cutover    # deploy HAProxy → App → Centrifugo
make smoke      # /healthz, /api/health, publish, RR distribution checks
make down       # destroy all 7

Three details matter.

(1) Terraform output → scripts/HOSTS.env auto-refresh. IPs are never hardcoded — not in the README, not in scripts. After make up, env vars like HAPROXY_IP, APP_IP, CENTRIFUGO_1_IP, … are populated automatically, and every downstream script starts with source scripts/HOSTS.env. IPs can change without a single line of code being touched.

(2) The secrets-check gate. If App and Centrifugo end up with different HMAC secrets, JWT verification silently fails everywhere. Hunting that down manually eats an hour. So before cutover, we automatically verify CENTRIFUGO_HMAC_SECRET_KEY matches between compose/.env.app and compose/.env.centrifugo.

secrets-check:
	@A=$$(grep '^CENTRIFUGO_HMAC_SECRET_KEY=' compose/.env.app);
	 B=$$(grep '^CENTRIFUGO_HMAC_SECRET_KEY=' compose/.env.centrifugo);
	 [ "$$A" = "$$B" ] || { echo "ERR: app/centrifugo HMAC mismatch"; exit 1; }

(3) PG data preservation (dump-pg / restore-pg). Vultr VMs are billed even in the stopped state, so the natural cycle is "tear down after a run, spin up next time." But that cycle is dangerous if it loses the PG seed in between.

make dump-pg           # dump current PG VM → data/postgres-dumps/
make list-dumps        # list saved dumps
make restore-pg        # auto-restore the latest dump to a fresh PG VM
make restore-pg DUMP=data/postgres-dumps/workb-XXXX.dump   # pin a specific dump

After restore, the script automatically prints ANALYZE results, table counts, DB size, and row counts for top tables — so nobody has to manually verify "did the restore actually work?"

Once wrapped this tightly, a new teammate or a future sprint can rebuild the environment with five commands. That's what "reusable" actually means here.

Observability — tuning without measuring is just guessing

Observability has to be alive before load goes in. Discovering breakage by running blind first is too late. So the monitoring stack was the first thing built, not the last.

Setup is three commands:

docker compose -f monitoring/docker-compose.monitor.yml up -d
./scripts/setup-monitoring-exporters.sh all
./scripts/verify-prometheus-targets.sh

setup-monitoring-exporters.sh is worth pointing out: it never puts PG/Redis passwords into the repo. The script SSHes into the remote VMs and reads /etc/postgres/postgres.env and /etc/redis/redis.env to build exporter-only env files there. There's no structural way for the secret to land in a commit.

The dashboard is split the way an operator actually thinks during a run — three layers:

(A) k6 / Sender         →  how much I tried to send
(B) Delivery / Receiver →  how much actually arrived
(C) Server / Resource   →  did the infra survive while that happened

Key panels per layer:

• (A) Sender — target TPS, achieved TPS, successful MPS, HTTP p50/p95/p99, 429 / 5xx split, k6 dropped iterations (how often constant-arrival-rate failed to keep up).

• (B) Delivery — REST-to-WS p50/p95/p99, receiver msgs/sec, active clients/subscriptions, expected traces (registered / delivered / pending), and anomalies (duplicate / parse failure / negative latency).

• (C) Server — node CPU/memory/network, Redis ops/memory/latency, Postgres connections/locks/deadlocks, Centrifugo per-node client/channel distribution, HAProxy 5xx by backend.

The value of this split is how fast you can isolate the cause. When 5xx spikes, if (B) latency is fine but (C) shows Postgres connections climbing — it's the DB pool. No guessing.

Auth / authorization seeding — the hardest part

This was the conceptually trickiest piece of the whole setup. It deserves its own section.

Why we bypass the login flow

The thing under test is POST /api/channels/{channelId}/messages — the write path. We're measuring the cost of: auth middleware → DB write → fanout → Centrifugo publish → WS delivery at the moment a message is sent.

If you include the full login flow (OAuth callback, session issuance, token exchange), the cost of the auth service blends with the message write path and you can't isolate where the knee actually is. So we preserve production's auth model exactly, but pre-seed everything that happens before the message is sent.

Two different auth models, kept distinct

The product has two client classes; the load test mirrors that.

The cost structures are completely different:

• REST auth: Redis GET + Postgres SELECT per message. Cost multiplies by message count.

• WS auth: one HMAC check at connect time. Zero cost afterwards.

That's why we never merge the two tokens. The production cost shape has to survive into the test.

"Seeding" means serializing the auth state

The word "serialize" is the right one. We're freezing into files the auth state that's normally scattered across memory, DB, and Redis. The state to freeze:

• Which users exist (User.id, email, name)

• Which workspace each user belongs to

• Which channels each user is a member of

• What access token to attach to each REST request

• Which Centrifugo channels each receiver may subscribe to

All of this is serialized into nine data/generated/*.jsonl files. These files are the SSOT of the test. Postgres, Redis, and k6 must all be looking at the same snapshot at the same time — drift between them produces an immediate flood of 401/403.

The generation pipeline

data/generated/users.jsonl  ← SSOT
        │
        ├──► seed-export-postgres.js
        │     → Postgres: upsert Workspace / User / WorkspaceMember /
        │                 Channel / ChannelMember
        │
        ├──► build-desktop-bearer-tokens.js
        │     → tokens.jsonl              (k6 attaches in Authorization)
        │     → redis-token-seed.jsonl    (input for Redis SETEX)
        │     → seed-redis-tokens.sh → Redis: desktop:token:access:*
        │
        └──► build-centrifugo-channel-allowlist.js
              → centrifugo-channel-allowlist.jsonl
              → mint-centrifugo-tokens.js  (HS256 sign with CENTRIFUGO_HMAC_SECRET_KEY)
              → centrifugo-jwts.jsonl     (Go receiver uses on WS connect)

Splitting the nine files into two groups by volatility turned out to be the operationally important part.

A. Auth-independent — reusable as long as the Postgres seed survives (6 files) users.jsonl, channels.jsonl, channel-members.jsonl, sender-channel-pairs.jsonl, mention-targets.jsonl, centrifugo-channel-allowlist.jsonl

B. Expiring / volatile — re-mint or re-seed every cycle (3 files) tokens.jsonl (1-year TTL, invalidated on Redis FLUSH), centrifugo-jwts.jsonl (1-year TTL, invalidated when HMAC secret rotates), redis-token-seed.jsonl (must be re-SET if Redis goes empty).

This grouping removes the recurring "do I have to regenerate everything?" question. As long as PG/Redis are alive and the tokens aren't expired, all nine files drop straight into the next 1k smoke or DB knee sweep.

Auth flows, end-to-end

REST sender:

k6 → Authorization: Bearer <opaque token>
   → Next.js auth middleware: Redis GET desktop:token:access:<token>
   → req.user injected
   → Postgres channel membership check
   → POST /api/channels/{id}/messages

WS receiver:

Go receiver → WS connect with token=<JWT>
           → Centrifugo: HS256 signature + exp check (no Redis/PG lookup)
           → subscribe chat:*

Pre-seeded auth is standard practice for message-path load tests. The goal isn't to benchmark the auth product itself — it's to find the knee in the message write path and fanout path.

Sender / receiver split measurement architecture

This is the core design choice of the load model.

[ client VM ]
   k6 (REST sender)  ──HTTP──►  HAProxy ──► App ──► (publish) ──► Centrifugo
                                                                      │
                                                                      ▼ (fanout)
   Go receiver (WS)  ◄──────────────────────────  Centrifugo ◄────────┘

Why send and receive don't live in the same process:

• k6 is optimized as an HTTP load generator. Bolting a WS subscriber onto it warps the generator-side load curve.

• The Go receiver holds 1,000+ concurrent WS connections and has to perform traceId matching + latency measurement per message. A goroutine/channel model fits naturally.

• Each side doing only its own job is what makes "which side is the bottleneck?" answerable.

Trace measurement — true REST → WS E2E latency

When k6 sends a message, it embeds metadata.clientTrace.traceId and sentAt in the payload. When the Go receiver gets a publication from Centrifugo, it matches by traceId and records now - sentAt.

sender POST timestamp ─────────────►  receiver WS receive timestamp
                  ▲                          ▲
            sentAt recorded            latency = now - sentAt

That value is the real REST-to-WS delivery latency p50/p95/p99 — not HTTP response time, but "how long from send until the message appears on someone else's screen."

Turn on EXPECTED_TRACE_MODE=direct and every successful send also registers its traceId at the receiver's /expected endpoint. The receiver now knows exactly which traceIds were supposed to arrive, which gives:

• registered — traceIds registered

• delivered — traceIds actually received

• pending — registered but never delivered (loss candidates)

• duplicate — same traceId received multiple times

• negativeLatency — clock skew suspects

At end of run, pending should converge to 0. Whatever's left is lost messages.

k6 scenario shape

export const options = {
  scenarios: {
    chat_smoke: {
      executor: 'constant-arrival-rate',  // arrival-rate based, not VU based
      rate: targetTps,
      timeUnit: '1s',
      duration,
      preAllocatedVUs,
      maxVUs,
    },
  },
  thresholds: {
    http_req_failed:       'rate<0.01',
    http_req_duration:     'p(95)<1000',
    dropped_iterations:    'count<1',
    message_send_429_total:'count<1',
    message_send_5xx_total:'count<1',
  },
};

constant-arrival-rate is the key. With VU-based scenarios, slower responses just lower the arrival rate, and you can't tell "did we fail to generate load, or did the server refuse it?" Arrival-rate-based means when the server can't keep up, dropped_iterations rises and the truth is visible.

The DB knee sweep takes this further: it auto-sweeps 50, 75, 100, 125, 150, 200, 250, 300, 400, 500, 650, 800, 1000 TPS with a 1-minute ramp + 5-minute plateau per stage. One run and the knee point is right there on the chart.

Execution flow — 12 fixed steps

For an environment to be genuinely reusable, the number of things humans have to remember has to be small. So a single load-test run is fixed into 12 ordered steps:

0.  Set working directory + source HOSTS.env
1.  Save existing PG dump          (make dump-pg)
2.  Provision 7-host infra          (make up → make ssh-test)
3.  Restore DB                      (make restore-pg)
4.  Deploy apps                     (make secrets-check → make cutover → make smoke)
5.  Bring up monitoring             (docker compose up + install exporters)
6.  Seed/export test data           (data:seed-postgres, data:validate)
7.  Refresh JWTs + Redis seed       (tokens:desktop-bearer, tokens:centrifugo)
8.  Upload artifacts to client VM   (upload-loadtest-data.sh, upload-k6-scripts.sh)
9.  Reset App connection pool       (docker restart witim-web)
10. k6 REST smoke probe             (chat-smoke.js, 30s)
11. Open the 3-terminal session     (receiver / Prom reverse tunnel / k6 sender)
12. Monitor abort criteria

Step 11 — the three-terminal pattern — is what gets used in every run:

• Terminal 1: Go receiver on the client VM — WS connections + trace matching + metric exporter.

• Terminal 2: SSH reverse tunnel so k6 on the client VM can reach the local Mac's Prometheus remote-write endpoint.

• Terminal 3: k6 sender on the client VM, streaming metrics out via Prometheus remote-write.

# Terminal 2: SSH reverse tunnel (k6 → Mac's Prometheus)
ssh -i "$SSH_KEY" -N -R localhost:9090:localhost:9090 "$SSH_USER@$CLIENT_IP"

The reverse tunnel is what makes k6 metrics show up in real time on the local Grafana dashboard. You watch the load break, in motion, and stop it the moment it does.

Abort criteria

If any of these become true, abort before overload and recovery curves bleed into each other:

message_send_5xx_total > 0
dropped_iterations > 0 (reproducibly)
HTTP p95 ≥ 1s
REST-to-WS p99 rising into seconds
Postgres connection exhaustion in logs
Postgres deadlocks repeating
client CPU pinned at 80–90% AND achieved TPS falling behind target

Why write these down ahead of time? Simple: if "should we keep going?" gets answered on gut feel mid-run, the resulting report loses credibility.

Reading a real run on Grafana — the 75→100 TPS knee

About 7–8 minutes into chat-knee-sweep.js, just as the sweep stage stepped from 75 TPS to 100 TPS, we caught this dashboard snapshot. What makes the moment interesting is that infra resources are barely touched, yet HTTP error rate spikes to 42.8% — a clear signal that the knee isn't a resource ceiling, it's something inside the app's processing path.

The dashboard is read in the three-layer order set up earlier: (A) k6 / Sender → (B) Delivery / Receiver → (C) Server / Resource.

(Continued — "I'll dig into the app server logs in the next post and unpack the cause of the 42.8% HTTP error rate*.)*

Thanks for reading. If you're running a similar measurement effort, the two patterns most worth taking from this post are probably "wrap IaC + deploy + smoke + seeding + teardown into one Make-driven flow" and "serialize the auth state into files and treat them as the SSOT." Those are the two things that decide whether the environment is reusable or just a one-shot.

I was building a real-time chat system designed to handle 100K concurrent WebSocket connections at 25K+ messages per second. The architecture was a symmetric dual-stack — two identical servers (16 vCPU / 58 GB each) running 5 Centrifugo nodes apiece, with Cloudflare DNS splitting traffic 50/50.

The critical challenge was cross-server message synchronization. When a user connected to Server A sends a message, it must reach a user on Server B. Centrifugo delegates this to Redis Pub/Sub — whichever Redis-compatible store sits behind it becomes the backbone of the entire messaging pipeline.

I chose KeyDB. It was a multithreaded Redis fork with active-replica mode, allowing bidirectional writes. Server A and B could both write, and changes would sync automatically. It looked perfect for a symmetric architecture.

Server A: KeyDB Primary         (read/write)
              ↕  (bidirectional replication)
Server B: KeyDB Active-Replica  (read/write)

Early load tests hit 23.6K TPS. Everything seemed fine.

Then I started reading GitHub Issues.

1. The Problem — 5 Years of the Same Bug

I began auditing KeyDB's issue tracker before production deployment. What I expected to find were old, resolved bugs. What I actually found was a pattern that spanned half a decade.

A Timeline of Hangs and Deadlocks

From 2019 to 2024, the same category of hang/deadlock kept recurring. And the latest version, v6.3.4, had not fixed any of them.

Structural Flaw: Multithreaded Deadlock

Issue #883 was the most alarming. It wasn't a simple bug — it was a structural flaw in KeyDB's multithreaded architecture itself.

bgsaveCommand attempts to acquire global WRITE lock
  ↓
AsyncWorkerQueue thread(1): waiting for global READ lock, holding m_mutex
  ↓
AsyncWorkerQueue thread(2): holding READ lock, waiting for m_mutex
  ↓
→ 3-way deadlock
  → CPU usage drops to 0%
  → All client connections unresponsive
  → Does not respond to SIGTERM (only SIGKILL works)

The entire server goes silent. No automatic recovery is possible. In Issue #845, users reported that even sudo reboot would hang — a hardware-level hard reboot was required.

A Project Effectively Abandoned

The last release, v6.3.4, was in October 2023. I was evaluating this in February 2026 — over two years with no updates. 233 open issues had accumulated on GitHub. Maintainer responses had virtually ceased since 2024.

I recalled what antirez, the creator of Redis, once said: "The risk of bugs in threaded code is very high, and Redis's non-threaded architecture is a design for stability."

2. Why This Is Catastrophic for Centrifugo

Centrifugo uses Redis Pub/Sub as the core path for inter-node message delivery. This isn't a cache layer. It's not a nice-to-have. If Pub/Sub freezes, every single Centrifugo node loses its ability to relay messages.

When KeyDB Pub/Sub freezes:
  ├─ Inter-node message delivery: completely halted
  ├─ All 10 Centrifugo instances: unable to deliver messages
  ├─ 100K connected users: total loss of real-time functionality
  └─ KeyDB ignores SIGTERM → no automatic recovery
      → An operator must manually hard-reboot the server

The Centrifugo documentation explicitly states: "KeyDB compatibility in future releases is not guaranteed." Every company running Centrifugo in production that I could find — VK, Badoo, ManyChat, Grafana — was using Redis or Redis + Sentinel. Not a single production deployment used KeyDB as a Pub/Sub broker.

3. Evaluating Alternatives — DragonflyDB vs. Valkey

KeyDB had to go. The question was what to replace it with.

DragonflyDB — Better Multithreading, But…

DragonflyDB is designed from the ground up as a multithreaded, shared-nothing architecture. Per-shard independent threads with minimal locking. The global lock contention that causes KeyDB's deadlocks simply cannot occur structurally. When deadlock issues are reported, a full-time team patches them within days to weeks.

But Centrifugo doesn't officially test against DragonflyDB. And there was a more fundamental question to ask.

Does Pub/Sub Even Need Multithreading?

For multithreading to provide any benefit, there's a prerequisite: a single core must be hitting 100%. The entire point of multithreading is distributing work across cores because one core can't keep up. So the real question is — when used as a Pub/Sub broker for Centrifugo, would Valkey's single-threaded design ever saturate one core?

The Centrifugo official benchmark shows that at 1 million WebSocket connections delivering 500K messages per second, each Redis instance used roughly 5% CPU. The developer himself stated: "I handled 500K connections with 10 Centrifugo nodes + 1 Redis instance, and Redis used only 60% of a single core."

Our environment is 100K connections. Proportionally, that's about 12–15% of a single core. On a 16 vCPU server, that's under 1% of total capacity. Pub/Sub simply relays messages — it's one of the lightest workloads Redis handles.

CPU load by Redis workload type:

GET/SET cache (hundreds of K ops/sec)   ████████████████  ← heavy
LPUSH/RPOP queue (bulk processing)      ██████████████    ← heavy
Sorted Set operations (ranking/analytics) ████████████    ← heavy
PUBLISH/SUBSCRIBE (message relay)       ███               ← light

But here's the critical insight. Let's say, for the sake of argument, that traffic spikes and Valkey's single core reaches 60%. Should the response be "switch to a multithreaded database"?

No. Just add another Valkey instance.

Centrifugo natively accepts Redis addresses as an array. When configured this way, it applies consistent hashing on channel names and automatically distributes traffic. This isn't Redis Cluster — the Valkey instances don't even know about each other. Centrifugo handles the routing: "this channel goes to A, that channel goes to B."

// Before: 1 Valkey instance — single core at 60%
{ "address": "redis://valkey-a:6379" }

// After: 2 Valkey instances — each core at 30%
{ "address": [
    "redis://valkey-a:6379",
    "redis://valkey-b:6379"
  ]
}

One line added to the config, and the load is halved. Three instances, one-third each. Linear horizontal scaling.

Here's what this means:

When is multithreading needed?
  = When a single core reaches 100%

What actually happens with a Pub/Sub broker:
  Valkey's single core reaches ~60%
  → Add one more Valkey instance, add one address to Centrifugo config
  → Each instance now handles ~30%
  → A single core never reaches 100%
  → The situation that requires multithreading simply never arrives

Instead of breaking the single-core ceiling with multithreading, you deploy multiple single-core instances. This is safer, more predictable, and the industry standard. And since Centrifugo supports this natively — there's no reason to use multithreaded Redis variants like KeyDB or DragonflyDB as a Pub/Sub broker.

The Verdict: Valkey

Valkey is a fork of Redis 7.2.4, led by the Linux Foundation with backing from AWS, Google, and Oracle. From Centrifugo's perspective, Valkey and Redis are indistinguishable — identical protocol, commands, Pub/Sub, and Sentinel support.

Being single-threaded means the entire bug category of "multithreaded deadlock" does not exist. If the top priority is "absolutely no freezing," Valkey + Sentinel was the most conservative and safest choice.

4. Building It — Valkey Primary + Replica + Sentinel (3-Node Quorum)

Architecture Change

I dropped KeyDB's active-replica (bidirectional writes) and switched to Valkey Primary → Replica unidirectional replication with Sentinel automatic failover.

Before (KeyDB):
  Server A: KeyDB Primary    ↔ (bidirectional)  Server B: KeyDB Active-Replica
  → Deadlock risk, no automatic recovery

After (Valkey + Sentinel):
  Server A: Valkey Primary   → (unidirectional)  Server B: Valkey Replica
  Sentinel ×3 (Server A, Server B, Runner) — quorum=2

  → Deadlock impossible, automatic failover in 5–15 seconds

Three Sentinels were placed on Server A, Server B, and a $6/month Runner server. With quorum=2, any single server failure still leaves two Sentinels to form a majority and trigger failover.

The Hard Part: Docker Networking vs. Sentinel

The most painful part of the migration wasn't Valkey itself — it was Docker networking.

Sentinel's job is to tell every node "here's where the current master is." But when Sentinel runs inside a Docker bridge network, it sees the master's address as 172.18.0.2 — a Docker-internal IP. When it passes this address to the Sentinel on Server B, that address either points to a completely different container or doesn't exist at all. Connection fails.

Sentinel inside Server-a's Docker bridge:
  "master is at 172.18.0.2!"
           ↓
Server-b's Docker bridge:
  "172.18.0.2? That doesn't exist here... → connection failed"

Think of it as giving someone a building's internal extension number and expecting it to work from a different building. Internal extensions only work within their own building.

The fix was running Sentinel with network_mode: host. When Sentinel sits directly on the host network instead of inside the Docker bridge, it sees the master at its VPC IP (10.10.0.3). VPC IPs are routable from any server in the network.

Sentinel (host network):
  "master is at 10.10.0.3:6379!"
           ↓
Server-b:
  "10.10.0.3? That's a VPC address — reachable! → connection success"

But there was another problem. Centrifugo containers still lived inside the Docker bridge, while Sentinel was now outside it. To reach from inside to outside, I needed the extra_hosts directive:

centrifugo-1:
  extra_hosts:
    - "sentinel:host-gateway"  # Route through bridge gateway to reach host's Sentinel

host-gateway is a special Docker keyword that resolves to the bridge network's gateway IP — essentially a lobby phone that connects the inside of the building to the outside. Finding this single line took longer than any other part of the migration.

Minimal Configuration Changes

No application code or Centrifugo configuration structure needed to change. One environment variable swap:

# Before
CENTRIFUGO_REDIS_ADDRESS=keydb:6379

# After
CENTRIFUGO_REDIS_ADDRESS=redis+sentinel://sentinel:26379,<server-b>:26379,<runner>:26379?master=mymaster

The address format changed to redis+sentinel://, and Centrifugo automatically queries Sentinel for the current Primary and connects. On the infrastructure side, I deleted three KeyDB-specific options (server-threads, server-thread-affinity, active-replica), replaced the KeyDB conf with a Valkey conf, and that was it.

5. Validation — Load Test Results

After migration, I ran the same stress test: 100K concurrent connections, 4-step progressive load.

TPS and Latency

The error-free sustainable ceiling was ~23.5K TPS. Through Step 3, the error rate remained at 0.0001–0.0002% — effectively zero.

Valkey and Sentinel Resource Usage

This is the answer to "isn't single-threaded a bottleneck?"

Valkey Primary reached 88–98% on its single core, but that's only ~6% of the 16 vCPU server's total capacity. The actual bottleneck was HAProxy at 216%. Sentinel overhead was negligible — 0.4–1.0% CPU and roughly 3.4 MB RAM.

Compared to the KeyDB era, removing bidirectional replication overhead noticeably reduced Replica-side CPU. And most importantly — we gained the structural guarantee that deadlocks cannot occur.

Failover Test

I forcefully killed the Primary and measured recovery time.

Total downtime: approximately 5–15 seconds. Under KeyDB, there was no automatic failover — an operator had to intervene manually. In the worst case, a hardware-level hard reboot was required. Now, a $6/month server provides fully automatic recovery.

Lessons Learned

The takeaways from this migration are clear.

Match the tool to the workload. Caches and queues are throughput-bound — multithreading has real benefits there. But a Pub/Sub broker is different. When a broker freezes, every connected client simultaneously loses real-time functionality. Retries are meaningless — the messages are already lost. For workloads where stability > performance, a multithreaded engine with deadlock risk is a liability, not an asset.

Single-threaded is not a weakness — it's a design decision. Redis and Valkey are single-threaded not because they gave up on performance, but because they eliminated the entire bug category of multithreaded deadlocks. When you need more throughput, add instances. It's safer, more predictable, and the industry standard.

GitHub Issues don't lie. I should have checked the Issues tab before choosing the stack. A project where the same bug recurs for five years, the last release was two years ago, and maintainers have stopped responding — putting that on a production critical path is a ticking time bomb.

Our platform, workb, had a straightforward meeting recording feature: hit record in the browser, stop when you're done, upload the file, click a button to transcribe. Simple — and deeply flawed.

Three pain points kept surfacing:

Browser memory blowup. The MediaRecorder API accumulates audio chunks as Blob/ArrayBuffer in memory. For a one-hour meeting recorded in WebM Opus, that's roughly 60–100 MB of raw chunks. When recording stops, new Blob(chunks) merges everything — and for a brief moment, both the original chunks and the merged blob coexist in memory, spiking to 120–200 MB. Add FormData conversion for upload and you're looking at 200–300 MB in a single tab's JS heap. Chrome enforces a per-tab memory ceiling of roughly 1–4 GB, and with 10+ tabs open, the GC starts thrashing, the UI janks, and the tab can outright crash. Worst part: a crash meant total data loss — nothing had been sent to the server yet.

Poor Korean STT quality. We were using OpenAI Whisper, which delivered around 85% accuracy on Korean speech — no speaker diarization, and a hard 25 MB file-size cap that made long meetings impossible to process.

Manual workflow friction. Users had to manually click "Transcribe" after recording. Forget to click? No transcript. Results were checked via polling every 5 seconds — 300+ redundant API calls over a typical 10–25 minute transcription window.

The New Architecture: A 6-Stage Automated Pipeline

We redesigned the entire flow into six stages that require zero user intervention after pressing "Stop":

1. Recording Capture — 5-minute savepoints with automatic cloud backup

2. Cloud Upload — Direct browser-to-R2 upload via presigned URLs

3. Auto-Trigger — Server-side transcription kickoff on finalize

4. Speech Recognition — VITO AI (sommers model) with speaker diarization

5. AI Summary — Claude API generates structured meeting minutes

6. Real-Time Delivery — Centrifugo WebSocket pushes status updates live

Let's walk through each stage.

Stage 1: 5-Minute Savepoints — Taming Browser Memory

The Core Idea

Instead of buffering an entire recording in memory, we flush audio to the cloud every 5 minutes. A 5-minute WebM Opus segment is only ~5–10 MB, so the browser never holds more than that at any point. One hour of recording? Still 10 MB in memory.

How It Works

1. Recording starts → the client sends an init request to the server and receives a recording ID + a presigned upload URL for the first part.

2. Every 5 minutes → accumulated audio chunks are bundled into a single blob and uploaded directly to Cloudflare R2. On success, the client notifies the server ("Part N saved"), receives the next presigned URL, and clears the chunks from memory.

3. Recording stops → the last partial segment is uploaded, and a finalize request tells the server to merge all parts.

Crash Recovery — Dual Mechanism

The biggest win is resilience. If the browser crashes mid-recording, every savepoint up to that moment is already safe in R2. Maximum data loss: 5 minutes.

We implemented two complementary recovery mechanisms:

Server-side timeout. Each savepoint schedules a "process automatically in 10 minutes" delayed job via BullMQ. When the next savepoint arrives, the previous timer is cancelled. If no savepoint shows up for 10 minutes, the server assumes a crash and proceeds to merge and transcribe whatever parts it has.

Client-side recovery UI. If the user reconnects, the app detects the interrupted session and displays: "Your previous recording was interrupted. Recover the saved 20 minutes?"

Why both? A single timeout creates a tradeoff between "safe wait time" and "fast recovery." A short timeout risks false positives from temporary network hiccups; a long one delays recovery for users who reconnect quickly. The dual approach lets us keep the timeout conservatively long while offering instant manual recovery.

The WebM Header Problem

An interesting gotcha: when we collected MediaRecorder chunks every 5 minutes and created a new Blob, the second segment onward was missing the WebM EBML header. ffmpeg refused to parse them.

The root cause is that MediaRecorder emits the container header only at the start of recording. Our solution: stop and restart the MediaRecorder on the same audio stream at each savepoint boundary. Each segment becomes an independent, fully valid WebM file. The resulting audio gap is 5–10 ms — completely imperceptible in a meeting context. This is a well-known pattern used by libraries like RecordRTC.

Stage 2: Direct-to-R2 Upload via Presigned URLs

In the old system, audio files passed through our application server on the way to storage. The new system eliminates the server from the data path entirely.

Presigned URLs are time-limited, pre-authenticated URLs that allow the browser to PUT directly to Cloudflare R2. The server only issues the URL (a lightweight operation involving signing, not data transfer). The actual file bytes never touch the server.

The result: zero server bandwidth consumption for uploads, regardless of how many users are recording simultaneously. At 100K users, the server's only job during recording is issuing URLs — a negligible cost.

Downloads work the same way: the server performs an auth check, logs the access, and redirects to a presigned download URL. R2 serves the file directly. Server memory usage for file I/O: zero.

We also discovered during code review that the R2 integration layer (provider class, presigned URL generation, storage key management) already existed in our codebase from a previous feature. We only needed to build the savepoint-specific API routes.

Stage 3: Automatic Transcription Trigger

Previously, users had to remember to click "Transcribe." If they forgot, no transcription happened — ever.

Now, when the server completes part merging via ffmpeg (the finalize step), it automatically enqueues a transcription job in BullMQ. No user action required. The old "Transcribe" button remains only as a manual retry option for failure cases.

Stage 4: STT Engine — From OpenAI Whisper to VITO

Evaluating 6 STT Engines

We benchmarked three commercial APIs (VITO, ElevenLabs, Google Chirp 3) and three open-source options (Whisper, Qwen3-ASR, Voxtral) as of March 2026.

Korean accuracy (Character Error Rate):

Speaker diarization: VITO (2–5 speakers, auto-detect), ElevenLabs (up to 32 speakers), and Google all have built-in support. Whisper requires external tooling like WhisperX.

VITO's differentiating features:

• ITN (Inverse Text Normalization): Converts spoken numbers to written form — "삼천원" → "3,000원."

• Disfluency filtering: Automatically removes filler words like "음," "어."

• Keyword boosting: Register up to 500 domain-specific terms to improve recognition accuracy.

None of the other engines include these features natively.

File size limits: Whisper's 25 MB cap was a hard blocker for long meetings. VITO Batch STT has no limit.

Cost Analysis by Usage Tier

Cost dynamics shift significantly across usage volumes (at $1 = ₩1,430):

Why We Chose VITO

For workb, Korean meeting transcript quality is the core product value. Our decision came down to three factors:

1. Best Korean accuracy at CER < 8%. In meeting transcription, lower accuracy doesn't just mean more errors — it means higher post-processing cost and user frustration.

2. All-in-one package. Speaker diarization + ITN + disfluency filtering are built-in. ElevenLabs has lower unit costs in some tiers but requires custom implementations for ITN and disfluency filtering. Google had reported stability issues with diarization.

3. Natural integration with our BullMQ architecture. We can pass the CalendarEvent attendee count as spk_count to improve diarization accuracy automatically.

We chose Batch STT over Streaming STT because batch mode processes the full audio file, producing significantly better speaker diarization. Streaming mode must make real-time speaker assignments without full context, leading to label swaps and split-speaker errors — unacceptable for multi-participant meeting minutes.

Streaming File Transfer: R2 → VITO

When feeding audio from R2 to VITO's API, we use stream piping instead of buffering the file into server memory:

The implementation uses https.request() + form.pipe() to stream directly from R2's response to VITO's upload endpoint. We couldn't use Node.js's native fetch for this because it doesn't support Readable stream bodies — a notable limitation that required dropping down to the https module.

This design lets us set BullMQ concurrency aggressively, independent of available server RAM.

Async Polling Without Blocking Workers

VITO's Batch API is asynchronous: you submit audio, receive a job ID, and must poll for completion. VITO doesn't support webhooks, so polling is the only option.

The naive approach — while (!done) { await sleep(30s); check(); } — locks a BullMQ worker slot for the entire 10–25 minute transcription duration. That's a severe bottleneck.

Our solution: BullMQ delayed jobs as a state machine.

1. submit-transcription job → sends file to VITO → returns immediately (slot freed).

2. Schedules a check-transcription job with a 30-second delay.

3. check-transcription runs → queries VITO once → if not done, schedules another check → returns immediately.

4. When complete → parses segments → enqueues AI summary job.

Each step occupies a worker for only a few seconds. 100 concurrent transcriptions? No problem — workers are never blocked.

Normalized Segment Storage

VITO's results are stored in two forms:

• segments column (JSON): Structured data with speaker ID, start/end timestamps in milliseconds, and text. Used for speaker filtering in the UI, timestamp-based playback, and AI summary input.

• transcript column (plain text): Human-readable format like [Speaker 1] 00:00:05 - Today's sprint review.... Used for full-text search, previews, and Claude API input.

We normalize VITO's field names into our own schema so that swapping the STT engine in the future won't break the downstream data contract.

Stage 5: AI Summary with Claude

The Summary Pipeline

Once VITO produces the transcript, Claude (claude-sonnet-4-20250514) generates structured meeting minutes. We feed it the transcript column (plain text) rather than the segments JSON — the summary only needs speaker labels and utterances, not millisecond-precision timestamps. Plain text also yields a more concise prompt and lower token usage.

Claude returns a JSON object containing: meeting title, duration, attendees, overall summary (3–10 sentences), per-agenda discussion points with decisions, action items (with owners and deadlines), and key insights.

Adaptive Tier System

Early on, we noticed that a fixed-length prompt produced oddly uniform summaries: a 10-minute standup and a 2-hour strategy session both got the same depth of coverage. Agenda discussion points were being truncated for longer meetings.

We introduced a three-tier system based on transcript length:

Stage 6: Real-Time Status via Centrifugo WebSocket

The Polling Problem

The old system used React Query to poll the server every 5 seconds: "Is it done yet?" Over a 10–25 minute transcription, that's ~300 wasted API calls.

Hybrid Approach: Push + Safety-Net Polling

The primary update channel is now Centrifugo WebSocket. Each pipeline stage (merging → transcribing → summarizing → complete) pushes a status update to the user in real time.

We still maintain a safety-net poll — but at 30-second intervals instead of 5. If the WebSocket connection drops, the client falls back to aggressive 5–15 second polling based on the current pipeline state.

Result: API calls dropped from ~300 to ~50 per transcription cycle.

Notification Routing

• Users viewing the recording detail page receive real-time UI updates via Centrifugo channel subscription.

• The person who initiated the recording receives a personal notification (toast + notification bell) regardless of where they are in the app — even in the desktop messenger client. Notifications are persisted in the database.

• Other meeting attendees are not notified in real time. They can access the completed transcript and summary through the meeting viewer.

The Common Design Philosophy

Looking across the three most impactful architectural decisions — R2 streaming pipe, 5-minute savepoints with presigned URLs, and BullMQ delayed-job polling — a shared principle emerges:

As concurrency scales, resource consumption stays nearly constant.

This isn't premature optimization — it's the difference between a system that falls over at 50 concurrent users and one that handles 100+ without architectural changes.

Tech Stack

Results

All six pipeline stages were completed and deployed as of March 25, 2026.

Memory stability: Browser memory stays under ~10 MB regardless of recording duration. Tab crashes now lose at most 5 minutes of audio.

STT quality: Korean recognition accuracy improved from ~85% to over 92%. Speaker diarization enables "who said what" — a feature that was previously impossible. No file-size limitations.

User experience: Transcription and summarization run automatically after recording stops. Real-time status updates replaced 300 polling calls with ~50. Zero manual steps required.

Server efficiency: File data never passes through the application server. Streaming transfer keeps server memory at ~KB even with 100 concurrent transcriptions. The architecture is designed to scale to 100K users.

This post documents the recording system overhaul for the workb platform. All benchmarks, cost figures, and architecture decisions reflect the state of things as of March 2026.