Postmortem: the thundering-herd cache collapse

Setup

A popular channel is served unicast from an edge that pulls from a single origin. Every few seconds a new live segment is produced. Players are roughly synchronised — they all want the newest segment at nearly the same instant.

Timeline

• T+0. A new segment becomes due. Tens of thousands of players request it in the same second.
• T+0. The segment is brand new — not yet in the edge cache. Every request is a cache miss.
• T+0.1s. The edge forwards all those misses to the origin at once — the thundering herd.
• T+1s. The origin, built to serve one edge not fifty thousand players, saturates. Requests queue and time out.
• T+2s. Players retry, adding load. The next segment is already due. The edge never catches up; playback fails widely.

Root cause

Not origin capacity in isolation — the failure was the absence of request coalescing. Nothing collapsed the flood of identical requests for one new segment into a single upstream fetch, so a predictable, synchronised demand turned into an origin-killing stampede. This is exactly why VOD CDN design fails for live.

The fixes

• Request coalescing at the edge: thousands of misses for the same segment become one origin fetch, fanned out to all waiters. This is the big one.
• Origin shield: a mid-tier that absorbs edge misses so the origin is never hit directly.
• Multicast on-net for STBs, which removes the per-viewer fetch entirely.
• Jittered client requests so players don’t all fire at the identical instant.

How HySky avoids it

HySky’s edge coalesces requests and sits behind an origin shield, and drives multicast for on-net STBs — so a synchronised segment boundary is absorbed at the edge instead of stampeding the origin.

Lessons

• Live segment boundaries create a synchronised, predictable request flood.
• Without request coalescing, cache misses stampede the origin.
• Coalescing + an origin shield turn the flood into a single fetch.
• Multicast removes the per-viewer fetch for on-net STBs entirely.