Rate limiting is one of those things that feels solved the moment you write the token bucket and watch the test pass. Mine passed every test I wrote for it. In production, behind Caddy, every single visitor to the site was sharing one bucket, ten requests a minute, for the entire site combined. Not per person. Total.
the limiter itself was never the bug
The actual bucket logic is a small, in-memory, per-key token bucket, nothing fancy, refills over time up to a burst equal to the per-minute rate:
func (l *Limiter) Allow(key string) bool {
l.mu.Lock()
defer l.mu.Unlock()
now := time.Now()
b, ok := l.buckets[key]
if !ok {
l.buckets[key] = &bucket{tokens: l.burst - 1, lastSeen: now}
return true
}
elapsed := now.Sub(b.lastSeen).Seconds()
b.tokens += elapsed * l.rate
if b.tokens > l.burst {
b.tokens = l.burst
}
b.lastSeen = now
if b.tokens < 1 {
return false
}
b.tokens--
return true
}That code is correct and hasn't changed. The bug was entirely in what gets passed in as key.
how I keyed it, and why it looked fine locally
The naive version keys purely on r.RemoteAddr, the raw socket address Go's own HTTP server sees for the connection:
// what I shipped first
func limiterKey(r *http.Request) string {
return r.RemoteAddr
}Locally, running everything through docker compose up on my laptop, that's exactly the browser's own address, 127.0.0.1 or close to it, one machine, one bucket, works exactly as expected in every manual test I ran.
The topology behind Caddy in an actual deployment isn't that. Every request to the blog service arrives through the frontend container acting as an internal reverse proxy, itself sitting behind Caddy, itself facing the internet. RemoteAddr at the blog service is the frontend container's address, for every single visitor, because that's genuinely who opened the TCP connection to the blog service from its point of view. A key based on that isn't per-visitor at all, it's per-hop, and there's exactly one hop between "every visitor on the internet" and "the blog service." A per-IP limiter built on that key is a de facto site-global limiter, and it stayed invisible in every local test because locally there's no proxy hop to hide behind in the first place, the bug only exists in the topology I never tested against directly.
the standard answer, and the trap inside it
The standard fix for "the address I see isn't the real client" is X-Forwarded-For, a header proxies append with the original client's address as they forward a request onward. Trusting it blindly is its own well-known trap though: X-Forwarded-For is a request header, and any client can set it to literally anything before Caddy ever sees the request, including a fake value chosen specifically to collide with someone else's rate-limit bucket, or to dodge their own. You can't just read it and trust it, you have to know which part of it, if any, your own infrastructure actually wrote.
The rule I landed on: trust exactly one value, and only when there's exactly one value to trust.
func clientIP(r *http.Request) string {
if xff := r.Header.Get("X-Forwarded-For"); xff != "" && !strings.Contains(xff, ",") {
if ip := strings.TrimSpace(xff); ip != "" {
return ip
}
}
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
return r.RemoteAddr
}
return host
}A single-value header means the blog service's own upstream (the frontend proxy, and only the frontend proxy) wrote it fresh. Anything with a comma in it, multiple hops chained together, is exactly what a client spoofing the header from outside would produce, since a real client has no earlier hop to have appended anything before their own fake value, so that shape falls back to RemoteAddr instead of being keyed on at all.
the other half of the fix lives one hop earlier
None of that works unless something upstream actually rewrites the header to a single trustworthy value instead of just relaying whatever the browser sent. That's the frontend proxy's job, and it does it deliberately, not by accident:
const inboundXff = event.request.headers.get('x-forwarded-for');
let clientAddress = '';
try {
clientAddress = event.getClientAddress();
} catch {
// no trustworthy client address for this request (e.g. a direct
// loopback healthcheck bypassing Caddy) - fall through to sending none
}
const forwardedFor = resolveForwardedFor(inboundXff, clientAddress);
if (forwardedFor) headers.set('x-forwarded-for', forwardedFor);resolveForwardedFor takes the last entry of whatever came in (the one Caddy itself appended, trustworthy regardless of what a client prepended earlier in the chain) or falls back to the frontend's own resolved client address, and always emits exactly one value onward. The blog service's single-value check only holds up because this earlier hop guarantees it, never forwarding a multi-value chain itself. Two services, two small pieces of logic, and the whole thing only actually works because each one trusts the layer immediately behind it and nothing further back than that.
what actually surfaced it
Not a report from an angry user getting rate-limited unfairly, which is what I expected would eventually flag this. It came up during a review pass while working through login and password-change endpoints, tracing through exactly who could see what address at each hop, and realizing the number "10 requests a minute" was true, just true for the wrong population. Once you say it out loud, "every visitor to the site shares one bucket," it sounds obviously broken. Getting there required actually drawing the request path hop by hop instead of trusting that a rate limiter which passed its unit tests was doing the job in the topology it would actually run in.
the same fix, twice
The blog service isn't the only place this exact shape of bug lived. The auth service keeps its own separate copy of the same limiter package rather than sharing a module with the blog service (this whole codebase duplicates a handful of small things like that on purpose, the JWT claims struct is another one, rather than pulling in a shared internal package for a dozen lines of code), and its login and password-change endpoints had the identical single-hop-behind-Caddy problem, keyed the same wrong way for the same reason. Fixing one and not the other would have meant one endpoint enforcing a real per-visitor limit while a sibling endpoint two services over stayed silently global. Once I understood the actual shape of the bug, checking whether it existed anywhere else the same request path touched took a few minutes. Writing this post took a lot longer than either fix did.
the boring lesson underneath the interesting one
A rate limiter with a correct token bucket and a wrong key isn't half-broken, it's fully broken, it just fails in a direction that looks like success from the outside, requests get allowed or denied, numbers move, nothing throws an error. That's the genuinely dangerous shape for this kind of bug to take. A limiter that crashed outright would have been caught the first time anyone hit the endpoint locally. One that quietly protects the wrong boundary keeps working, right up until the boundary it was actually supposed to protect gets hit by someone who isn't playing nice.