Postmortem: the big-match entitlement storm

Setup

A platform authorises each playback by calling a central entitlement service synchronously, which checks a database. At normal concurrency this is fine — a few hundred checks a second, well within the connection pool. Then a marquee match kicks off.

Timeline

• T+0s. Kickoff. Tens of thousands of set-top boxes and apps request the same channel within the same minute — a synchronised authorisation spike.
• T+20s. The central auth service’s request queue grows faster than it drains. Each check holds a database connection while it waits.
• T+35s. The database connection pool is exhausted. New auth checks now wait for a free connection — latency climbs from milliseconds to seconds.
• T+50s. Players time out waiting for authorisation and retry. Retries add load to an already-saturated service — a retry storm.
• T+70s. The amplification tips it over: the auth service is spending all its time on connections that will time out anyway. Playback fails broadly, even for users whose entitlements were perfectly valid.

Root cause

Not capacity in the raw sense — the failure was architectural. A synchronous, central, per-playback entitlement check with no caching turned a predictable concurrency spike into a queue; a finite connection pool turned the queue into exhaustion; and naive client retries turned exhaustion into a self-amplifying storm.

The fixes

• Edge token validation. Authorise playback locally against a signed token and a read-replica, so the spike never hits one central service. This is the big one.
• Pre-authorisation / caching. A valid entitlement doesn’t need a fresh central check on every single play.
• Connection pooling + backpressure. Bound the work and shed load gracefully instead of collapsing.
• Jittered, capped retries. Clients must not retry in lockstep; add jitter and a ceiling.
• Capacity for peak, not average. Size for the big match, because the big match is the whole point.

How HySky avoids it by design

HySky pushes entitlements to the edge as HMAC tokens validated locally, so a synchronised big-match spike is absorbed at the edge instead of hair-pinning into one central authoriser. There is no single entitlement service for the storm to saturate — the same design that keeps a dead node from becoming an outage.

Lessons

• Synchronised concurrency (a big match) is a predictable load — design for it.
• A central synchronous entitlement check + finite pool + naive retries is a storm generator.
• Validate entitlements at the edge with tokens; don’t hair-pin every play.
• Add jittered, capped retries and real backpressure — and size for peak, not average.