Use case · Social & messaging
News feed celebrity fan-out
Copying every new post into each follower's feed is fast, until one account has ten million followers. Watch one post clog the queue, then pull it at read time.
Used by apps like X, Instagram, Facebook, LinkedIn and Threads
Happy path · break it · fix it
News feed celebrity fan-out: sequence diagram
News feed celebrity fan-out, shown as messages between Poster (The app of someone publishing a post: an ordinary user or a celebrity), Posts (The post service, which stores every post once, by author), Fan-out (The fan-out queue and its workers, which copy a new post's id into followers' feeds), Feed cache (The feed service and its cache: one ready-made list of post ids per reader), Reader (The app of a follower opening their home feed).
Happy path: 8 steps.
- Poster → Posts: POST /post. An ordinary user with 300 followers publishes a post.
- Posts: post p1 stored. The post itself is stored once, under its author.
- Posts → Poster: 201 posted (reply). The poster gets an answer straight away. The rest happens in the background.
- Posts → Fan-out: job p1. One fan-out job is queued: add this post to the feed of each of the 300 followers.
- Fan-out → Feed cache: add p1 (many at once). Workers write the post's id into every follower's cached feed. It is done in a moment.
- Feed cache: feeds ready to read. Each follower's feed is now a finished list, built before anyone asks for it.
- Reader → Feed cache: GET /feed. A follower opens the app and asks for their home feed.
- Feed cache → Reader: feed + p1 (reply). One read returns the ready-made feed, new post included. Reads stay cheap.
Result: Work is done once at post time, so every read is one lookup.
Break it: 10 steps.
- Poster → Posts: POST /post. This time the poster is a celebrity with 10 million followers (an illustrative number).
- Posts → Poster: 201 posted (reply). The post is stored once and the celebrity gets the usual quick answer.
- Posts → Fan-out: job c1. The same code path queues the same kind of job, but this one means ten million feed writes.
- Fan-out → Feed cache: add c1 (many at once). Every worker starts copying the same post into feed after feed. It will take minutes.
- Fan-out: queue backs up. Writes arrive far faster than workers finish them, so the fan-out queue grows.
- Poster → Posts: POST /post. A moment later an ordinary user posts to their 300 followers, as in the happy path.
- Posts → Fan-out: job p1. Their small job joins the back of the same queue. Feeds are meant to be fresh within 5 seconds.
- Fan-out → Feed cache: add p1 (never arrives). It waits behind millions of writes for someone else's post, and the 5 seconds run out.
- Reader → Feed cache: GET /feed (many at once). The ordinary user's friends open the app and pull to refresh.
- Feed cache → Reader: no p1 yet (reply) (many at once). Their cached feeds are stale: the friend's post is stored, yet nobody can see it.
Result: One famous account made everybody's feed late.
Fix it: 10 steps.
- Poster → Posts: POST /post. The same celebrity with 10 million followers publishes the same post.
- Posts: 10M > limit: no job. Over the follower limit: the post is stored once and marked for pulling at read time.
- Posts → Poster: 201 posted (reply). The celebrity gets the same quick answer. Nothing was added to the fan-out queue.
- Poster → Posts: POST /post. A moment later the ordinary user posts to their 300 followers.
- Posts → Fan-out: job p1. Under the limit, so it fans out as before, and the queue in front of it is empty.
- Fan-out → Feed cache: add p1 (many at once). The friends' cached feeds get the post within the 5 seconds. The queue stays short.
- Reader → Feed cache: GET /feed. A reader opens the app. Their cached feed holds posts from ordinary accounts only.
- Feed cache → Posts: new from my celebs?. The feed service also asks for recent posts by the few celebrities this reader follows.
- Posts → Feed cache: c1 (reply). A handful of short lookups, one per celebrity followed, and easy to cache.
- Feed cache → Reader: p1 + c1 (reply). The two lists are merged by time: the friend's post and the celebrity's post both appear.
Result: Ten million writes became a few reads, only for readers who show up.
step 1 of 8
An ordinary user with 300 followers publishes a post.
- request
- success
- held / reserved
At scale
Every dot is one feed write. Ordinary users post at a steady pace; three seconds in, the account in the middle posts. Pick how many followers it has. Numbers are illustrative and not to scale.
Numbers are illustrative: a small simulation, not a measurement.
The happy path, step by step
- A user publishes a post from the app.
- The post service stores the post once, under its author, and answers the poster straight away.
- It queues one fan-out job for the post.
- Fan-out workers look up the author's followers and add the post's id to each follower's cached feed.
- For an account with a few hundred followers that is a few hundred small writes, finished in a moment.
- Every follower's feed is now a ready-made list, built before anyone asks for it.
- A follower opens the app and asks for their home feed.
- The feed service returns the cached list in one read, with the new post in it.
The failure
What breaks and why
Fan-out on write does work in proportion to the author's follower count, and follower counts are wildly uneven. A post from an account with millions of followers goes down the same code path and becomes millions of feed writes, far more than the workers finish per second, so the fan-out queue backs up. The queue is shared, so an ordinary user's small job posted a moment later waits behind someone else's millions of writes, and their friends refresh and see nothing new. Most of that work is also wasted: many of those followers will not open the app before the post is old.
Before / after
The fix
Use both models and choose per author. Posts from ordinary accounts are still pushed into followers' cached feeds, which keeps reads cheap. An author over a follower limit is stored once and never fanned out. When a reader opens the app, the feed service reads the cached feed, fetches recent posts from the few very large accounts that reader follows, and merges the two by time. The cost moves from millions of writes at post time to a few cacheable reads, paid only for readers who turn up. The numbers and the limit here are illustrative.
on_post(post):
store(post)
for follower in followers(post.author):
push(feed[follower], post.id) // 10M writes
on_read(reader):
return feed[reader]on_post(post):
store(post)
if follower_count(post.author) > LIMIT:
return // pull at read
for follower in followers(post.author):
push(feed[follower], post.id)
on_read(reader):
pushed = feed[reader]
pulled = [recent_posts(c)
for c in big_accounts_followed(reader)]
return merge_by_time(pushed, pulled)From their own write-ups
How real companies approach it
LinkedIn says it weighed both models when it built FollowFeed and chose fan-out on read: a member's feed is put together on request from the timelines of the entities they follow. It reports that in its experiments precomputed feeds needed 62 times more data, and that they made new ranking models slower to test.
LinkedIn Engineering: FollowFeed: LinkedIn's Feed Made Faster and Smarter ↗Facebook describes Multifeed, a backend behind News Feed, as doing its work at read time: when someone opens their feed it looks up their friends, finds their recent actions and ranks them. An aggregator sends each query to leaf servers that keep recent actions in memory.
Engineering at Meta: Serving Facebook Multifeed: efficiency, performance gains through redesign ↗Facebook says load spikes are common on TAO, its graph store: they happen when a handful of objects become extremely popular by appearing in the News Feeds of tens of millions of people at once. It describes moving or cloning shards between servers to even out that load.
Engineering at Meta: TAO: The power of the graph ↗
How this comes up in interviews
"Design the home feed for a social network. What happens when an account with millions of followers posts?"
Then they ask
- Where do you set the follower limit between push and pull, and what happens to an account that crosses it?
- A reader follows many very large accounts. How do you keep that reader's feed fast?
- Ordinary users' posts arrive late while one large fan-out is running. How do you isolate them?
A strong answer covers
- Fan-out on write makes a feed read one cached list, but its work is in proportion to the author's follower count.
- A post from a huge account becomes millions of feed writes, backs up the shared queue and delays everyone else's posts.
- Choose per author: push for ordinary accounts, store posts from accounts over a follower limit once and never fan them out.
- At read time, merge the cached feed with recent posts from the few very large accounts the reader follows, by time.
- The cost moves to a few cacheable reads, paid only for readers who turn up.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
Your turn
What would you do?
Your feed copies each new post into every follower's cached feed through one worker queue. An account with 12 million followers posts, the queue grows for minutes, and ordinary users' posts reach their friends late. Which change removes the cause?
Related
News feed celebrity fan-out questions
- What is the difference between fan-out on write and fan-out on read?
- Fan-out on write (push) copies each new post into every follower's stored feed when it is published, so reading a feed is a single lookup. Fan-out on read (pull) stores each post once and builds the feed when a reader asks, by fetching recent posts from everyone they follow. Push makes reads cheap and writes costly; pull does the opposite.
- What is the celebrity problem in news feed design?
- With fan-out on write, one post costs one write per follower. For an account with millions of followers a single post becomes millions of writes, which can back up the fan-out queue and delay everyone else's posts. The usual answer is a hybrid: push for ordinary accounts, and pull the posts of very large accounts when a feed is read.
- Is this the same as Instagram's "Justin Bieber problem"?
- It is a cousin of it. Wired reported in 2015 that so many likes on one celebrity's photos overloaded Instagram's cache and database, and that Instagram engineers described moving to a like count kept in a single database cell. That is a hot counter, not feed fan-out, but the root is the same: one account far larger than the design assumed.
Sources: LinkedIn Engineering: FollowFeed: LinkedIn's Feed Made Faster and Smarter ↗ · Engineering at Meta: Serving Facebook Multifeed: efficiency, performance gains through redesign ↗ · Engineering at Meta: TAO: The power of the graph ↗ · Wired: How Instagram Solved Its Justin Bieber Problem (press reporting, 2015) ↗ · Report a problem
A typical design, informed by public write-ups, not any one company's exact system. Diagrams are our own. Numbers in the simulation are illustrative. All wording is original and paraphrases the linked sources.