Until this week, when something went wrong on this blog the evidence lived in docker logs on the server, which means it effectively lived nowhere. A user could hit an error, tell me about it, and my debugging tool was scrolling terminal output looking for a timestamp that roughly matched their message. Time to fix that.
The idea is simple: every time a service writes an error response, it also writes a row to an error_log table. Path, status, error code, message, request id, the user if there was one. The admin panel gets a logs tab that reads from both services and merges them into one list I can filter. User says "it broke around eight", I filter to that window and see exactly what their request hit.
The first design decision that turned out wrong: I only logged 5xx errors, on the theory that 4xx means the user did something wrong and 5xx means I did. But 4xx responses are often the clue. A pile of 413s means someone keeps hitting an upload limit I set too low. A stream of 403s means either an attack or a permissions bug. So the net widened to 5xx plus a curated few: 403, 413 and 429. Not every 404 from a bot scanning for wordpress.php, that would just be noise with a database bill.
The subtle bug in the middle of this one was about context. The obvious code passes the request's context into the database insert. But by the time the error response is written, that request is basically over, and its context can already be cancelled, which kills the insert. The log row about the failure gets lost because the failure happened. The fix is to give the insert its own detached context with a short timeout:
ctx, cancel := context.WithTimeout(context.Background(), errorLogInsertTimeout)
defer cancel()
if err := q.InsertErrorLog(ctx, db.InsertErrorLogParams{ ... }); err != nil {
log.Printf("error_log insert failed: %v", err)
}And note what happens when the insert itself fails: a plain log line, nothing more. The one thing an error logger must never do is produce an error response of its own, because that error would get logged, which could fail, which would get logged. I have read enough postmortems to be properly scared of that loop.
The tab also got a delete button per row and a clear-all, because a log you cannot clean up just becomes a wall of old noise you stop reading. Rows older than 90 days sweep themselves out along with the analytics events, same janitor, same schedule.
Is a two-table error log with a merge in the browser a grand observability platform? No. There is a proper central logging service sketched out for v2, with batching and retry when the network drops. But that is a design for later. What I needed this week was to stop debugging through docker logs, and that problem is now solved with two tables and one admin tab.