Thundering Herd: The Problem That Kills Ruby Apps at Scale

Picture this.

You just shipped that fancy new report page. Stakeholders are thrilled, product is spamming emoji reactions in Slack, and you’re feeling pretty good about your deployment. You added caching, deployed like a pro, and went for coffee.

Then the error reports start flooding the #errors channel.

Your manager DMs you:

“Why are we seeing 500 errors?”

Your cache expires… One request comes in, misses, and quietly goes off to the DB to rebuild the data. No big deal, right?

Now imagine a thousand requests hitting that same key in the exact same millisecond. Every single one sees an empty cache. Every single one fires off the same expensive query. Your database gets absolutely dogpiled. Response times explode, the connection pool melts, and users start seeing timeouts and 500s.

This is the thundering herd, also known as a cache stampede.

This is a classic one.

In this post, I am going to break it down step by step, show you exactly how it plays out in a real app, and give you a couple of practical fixes you can actually use. Whether you’re just starting out or you’ve been doing this for years, there’s something here for you. And if you want to get your hands dirty right now, I threw together a little repo so you can reproduce the stampede yourself and play with the fixes: https://github.com/wh1le/ruby-at-scale and then come back to read the rest of this post.

Why This Matters Across Experience Levels

If you’re early in your career: You probably haven’t run into this one yet. Most tutorials just teach you “cache the expensive thing” and move on. Nobody ever mentions what happens the moment that cache key expires under real load. Grasping this early gives you a distinct advantage, allowing you to design systems that handle production-level traffic safely from day one. Sure, for heavy computational reports you can sometimes lean on materialized views in SQL and call it a day. But for everything else that doesn’t live in the database (complex business logic, external API calls, custom aggregations, etc.) you’re on your own with custom cache runners in Ruby. That’s exactly where the thundering herd sneaks up and bites you.

If you’re a senior engineer: You know how to cache, but you might not realize how heavily concurrent traffic spikes transform minor I/O hitches into cascading infrastructure failures - especially when high-cost LLM API billing is tied to every cache miss.

The Anatomy

To understand why this happens, let’s look at the standard, naive approach to caching that most of us write without a second thought:

def fetch_from_cache(key, ttl: 300)
  value = redis.get(key)
 
  if value.present?
    return normalize(value)
  else
    value = compute(yield)
    redis.set(key, value, ttl: ttl)
  end
 
  return value
end
 
# app/controllers/products_controller.rb
# This endpoint is hit by thousands of public shoppers visiting a merchant's store
 
def index
  # During a flash sale, 1,000 shoppers hit this at once.
  # If the cache is empty, all 1,000 threads will run this 2.5-second 
  # LLM/DB query simultaneously.
  cache_key = "feeds:hot_deals:merchant:#{current_merchant.id}"
 
  @feed = fetch_from_cache(cache_key) do
    LLMProductFeedHotDealsQuery.call(merchant_id: current_merchant.id)
  end
 
  render json: @feed
end

Here’s what goes wrong step by step:

1. Cache key feeds:hot_deals:merchant:42 has a TTL of 300 seconds and lives in Redis.
2. At T=300s, the key expires.
3. Request A arrives, misses the cache, and starts the heavy 2.5-second LLM query.
4. Request B arrives 1ms later, checks cache, still a miss (A hasn’t finished yet). Also queries the LLM query.
5. Requests C through Z (and thousands of others) all pile in during the 2.5 seconds it takes to rebuild

The query itself takes 2.5 seconds. Painful for one user, but manageable. Catastrophic when multiplied by thousands of shoppers hitting a flash sale.

T=300.000s  Cache expires
T=300.001s  Request A → cache miss → starts 2.5s LLM query
T=300.002s  Request B → cache miss → starts 2.5s LLM query
T=300.003s  Request C → cache miss → starts 2.5s LLM query
...
T=302.499s  Request Z → cache miss → starts 2.5s LLM query
T=302.501s  Request A finally finishes, writes payload to Redis
T=302.502s  Request Z+1 → cache HIT (finally)

With 1,000 requests hammering your storefront every second, you just triggered 2,500 identical, concurrent LLM and database connections. Your database connection pool maxes out instantly. Your Ruby workers clog up waiting for API responses. Incoming traffic backs up, your server memory spikes, and your users face a wall of 504 Gateway Timeouts.

The really insidious part: this is invisible at low traffic. Your staging environment with a few testing requests will never trigger it. It only shows up in production, under real load, at the worst possible moment. And because you’re throwing duplicate requests at an AI model instead of a standard database? Imagine the bill you get charged for LLM compute.

The obvious answer is “make the cache last longer.” But that’s not a solution, it’s avoidance.

Your data has freshness requirements. Product prices change. Inventory updates.

A longer TTL also means: Users see stale data for longer. When the cache does finally expire, the stampede is even worse because traffic has grown during that longer window.

You haven’t solved the problem. You’ve just made it less frequent and more severe when it hits.

Common Solutions (and Their Tradeoffs)

1. Mutex Lock (Single Recompute)

Only one request rebuilds the cache. Everyone else waits or gets stale data.

The idea: Acquire a distributed lock (Redis SET NX). Winner rebuilds. Others either wait and retry, or return stale data.

This is the actual solution from the ruby-at-scale repo:

class Cache
  LOCK_TTL = 15
  POLL_INTERVAL = 0.05
 
  def initialize(redis = Redis.new)
    @redis = redis
  end
 
  def fetch(key, ttl: 60)
    cached = cached_value(key)
    lock_key = "#{key}:lock"
 
    return cached unless cached.to_s.empty?
 
    if lock!(lock_key)
      result = yield
      redis.set(key, result, ex: ttl)
      redis.del(lock_key)
      result
    else
      await_redis_cache(key)
    end
  end
 
  private
 
  attr_reader :redis
 
  def await_redis_cache(key)
    deadline = waiting_deadline
 
    loop do
      sleep(POLL_INTERVAL)
      value = redis.get(key)
      return value if value
      break if Time.now > deadline
    end
  end
 
  def waiting_deadline
    Time.now + LOCK_TTL
  end
 
  def lock!(lock_key)
    redis.set(lock_key, true, nx: true, ex: LOCK_TTL)
  end
 
  def cached_value(key)
    redis.get(key)
  end
end

Let’s break down what’s happening:

1. cached_value(key) - check Redis for existing data. If it’s there, return immediately. No lock needed.
2. lock!(lock_key) - attempt SET NX (set if not exists) with a TTL. This is atomic in Redis. Out of 1000 concurrent requests, exactly one gets true.
3. The winner executes the block, writes the result to Redis, and releases the lock.
4. Everyone else enters await_redis_cache - a polling loop that sleeps 50ms between checks until the value appears or the deadline passes.

Tradeoff: What if the winner crashes mid-rebuild? Now everyone is waiting on a lock that never releases. You need lock expiry (the ex: LOCK_TTL above). But if the rebuild legitimately takes longer than the lock TTL… you’re back to stampede territory. You’re also adding latency for the “others” - they’re polling every 50ms instead of getting an instant response.

When to use it: This is the workhorse solution. It works for most cases where the rebuild time is predictable and shorter than your lock TTL. The repo challenge uses this approach because it’s the most universally applicable.

2. Probabilistic Early Expiration

Randomly refresh the cache before it actually expires. Each request has a small probability of triggering a refresh when the key is “close” to expiring - the closer to expiry, the higher the probability.

This approach is described in detail on Wikipedia.

Tradeoff: Elegant but harder to reason about. Works great at scale (with thousands of requests, someone will refresh early), but unreliable on low-traffic endpoints where you might get unlucky and nobody refreshes before expiry.

When to use it: High-traffic keys where you want zero-downtime refreshes and can tolerate occasional redundant recomputes.

3. Background Refresh

Never let the cache expire. A background job refreshes it proactively.

The idea: Sidekiq job runs every 60 seconds to rebuild a cache with a 90-second TTL. The overlap ensures the cache never goes cold between job executions.

class CacheRefreshJob
  include Sidekiq::Job
 
  def perform(key)
    result = ExpensiveQuery.run
    REDIS.set(key, serialize(result), ex: 90)
  end
end
 
# Scheduled every 60 seconds
Sidekiq::Cron::Job.create(
  name: 'refresh_popular_products',
  cron: '*/1 * * * *',
  class: 'CacheRefreshJob',
  args: ['popular_products']
)

Tradeoff: Works perfectly for predictable, known cache keys. Falls apart when you have dynamic keys (per-user, per-query). You can’t background-refresh keys you don’t know about in advance. Also adds operational complexity - now you have a job that must never fail, or your cache goes cold anyway.

When to use it: Dashboard data, leaderboards, aggregate stats - anything with a known, finite set of cache keys that are always needed.

The Solution in the Repo

The ruby-at-scale challenge uses the mutex lock approach. Here’s why: it’s the most general-purpose solution, it works with any cache key (dynamic or static), and it demonstrates the core concurrency concept clearly.

The setup is a Sinatra app running on Puma with 4 workers and 5 threads each - 20 concurrent request handlers. The test fires 1000 curl requests simultaneously and checks that stampede:query_count in Redis equals 1. If your implementation works, only one process executes the expensive query. The other 999 wait and get the cached result.

# config.ru - the endpoint under test
get '/report' do
  result = RubyAtScale::CacheStampede.cache.fetch('expensive_report', ttl: 60) do
    RubyAtScale::CacheStampede.redis.incr('stampede:query_count')
    RubyAtScale::CacheStampede.expensive_query
  end
  { pid: Process.pid, result_size: result.to_s.length }.to_json
end

The expensive_query simulates a slow DB call. The incr('stampede:query_count') is the tripwire - if it’s anything other than 1 after the test, you have a stampede.

The key insight: Redis SET NX (set if not exists) is atomic. In a world of 20 concurrent threads across 4 processes, exactly one will win the lock. The rest see false from SET NX and know someone else is handling it. They poll until the value appears.

Why I Built ruby-at-scale

I kept running into the same pattern: developers who could explain caching in theory but froze when asked “okay, but what happens under concurrent load?” in interviews. Or engineers who’d been writing Rails for years but never had to think about this because their traffic never demanded it.

Most learning resources teach you the happy path. They show you Rails.cache.fetch and call it a day. The real learning happens when things break under pressure and you can’t simulate that by reading a blog post.

I wanted something where you actually feel the problem. You run the test, you see “STAMPEDE: 47 total queries hit the DB over the execution window”, and realize your database is taking a massive hit. Then you implement the fix, run it again, and see ”✓ Cache stampede prevented.” That feedback loop is how the concept sticks.

So I built a repo where you actually implement the solution. You get a broken cache, a test that simulates concurrent requests, and you have to make it work. No hand-holding, no step-by-step tutorial - just a problem, constraints, and a verification script.

Try It Yourself

The challenge is live: ruby-at-scale on GitHub

1. Clone the repo
2. Run bundle install and bin/setup
3. Read the challenge in lib/ruby_at_scale/cache_stampede/cache.rb
4. Implement your fix in the fetch method
5. Run the test: spec/system/bin/cache_stampede_test

You’ll need PostgreSQL and Redis running locally. The README has setup instructions. If you’re using Nix, there’s a flake.nix that sets up the full environment.

The test script:

• Clears the relevant Redis keys
• Starts Puma (4 workers, 5 threads each)
• Fires 1000 concurrent curl requests at /report
• Checks stampede:query_count - expects exactly 1

The test is opinionated: it doesn’t care which approach you use. Mutex, probabilistic, background refresh - whatever. It only cares that under 1000 concurrent requests with an expired cache, your database sees exactly one query. Not 47. Not 3. One.

The skeleton gives you LOCK_TTL = 15 and POLL_INTERVAL = 0.05 as constants. That’s a hint toward the mutex approach, but you’re free to ignore them entirely and do something different.

Full code and challenges: ruby-at-scale

All articles in this series: Ruby at Scale

Next in the series: Boundary Burst: The Rate Limit Problem Nobody Catches

---