So I got tired of opening four different sites every morning just to see if anything was actually new, so now a script does that part for me and emails me a summary at 7am. Mostly. There was a stretch of about two weeks where it silently did nothing at all and I didn't notice, which is its own story.
what it's supposed to pull
A handful of sources (Hacker News's API, NRK's RSS feed, a couple others), fetched, parsed, then filtered against a list of keywords I actually care about. Everything else gets thrown away, which is most of it. That's kind of the whole point.
import requests
KEYWORDS = ["postgres", "sveltekit", "linux", "norway", "helgeland"]
def fetch_hn_headlines():
ids = requests.get("https://hacker-news.firebaseio.com/v0/topstories.json").json()[:50]
headlines = []
for story_id in ids:
item = requests.get(f"https://hacker-news.firebaseio.com/v0/item/{story_id}.json").json()
title = item["title"]
if any(k in title.lower() for k in KEYWORDS):
headlines.append((title, item["url"]))
return headlinesThat version ran fine for about a week. Then one morning, no digest.
the two weeks I didn't notice anything was wrong
Cron doesn't email you when a script fails unless you've set MAILTO, which I hadn't, so a crash three lines into a for-loop just silently ends the job and nothing tells you. I found out purely by accident, mentioning to someone that the digest was handy, then realizing I hadn't actually seen one in a while and going to check.
crontab -l0 7 * * * /home/bjorn/.venv/bin/python /home/bjorn/digest.pyNothing there points at the actual problem, it just says the job exists. I had to run it by hand to see anything.
python3 digest.pyTraceback (most recent call last):
File "digest.py", line 12, in fetch_hn_headlines
title = item["title"]
KeyError: 'title'what I actually googled, and what I found
"python requests item some ids missing fields hacker news api" got me to a couple of Stack Overflow threads about the HN API specifically, and the actual answer was almost boring once I saw it: a "dead" or deleted story still shows up in the topstories list, but its item lookup comes back with no title field at all, sometimes not even a real object, just None. My code assumed every ID in that list points at a normal, complete story. It doesn't. Somewhere in that morning's fifty IDs was one dead one, and item["title"] blew up the entire loop, which killed the whole script, which meant zero headlines got fetched for any source that morning, or any morning after, since the crash happened before anything got sent and cron just quietly ate the failure every single day after that.
The accepted answer on the thread that actually helped wasn't about the HN API specifically, it was a general pattern: never let one bad item in a batch take down the whole batch. Wrap the per-item work, log what failed, keep going.
the version that actually survives a bad item
import logging
import requests
logging.basicConfig(filename="/home/bjorn/digest.log", level=logging.WARNING)
KEYWORDS = ["postgres", "sveltekit", "linux", "norway", "helgeland"]
def fetch_hn_headlines():
ids = requests.get("https://hacker-news.firebaseio.com/v0/topstories.json", timeout=10).json()[:50]
headlines = []
for story_id in ids:
try:
item = requests.get(
f"https://hacker-news.firebaseio.com/v0/item/{story_id}.json", timeout=10
).json()
title = item.get("title")
if not title:
continue
except (requests.RequestException, AttributeError) as e:
logging.warning("skipping story %s: %s", story_id, e)
continue
if any(k in title.lower() for k in KEYWORDS):
headlines.append((title, item.get("url", "")))
return headlinesOne bad item now gets logged and skipped instead of taking the whole run down with it. I also added MAILTO to the crontab, so a future crash actually reaches my inbox instead of vanishing into a log file I have to remember to check.
reusing the email code
The email-sending half is basically lifted straight from the camera project, just without the image attachment. Once you've written smtplib boilerplate once, you don't really want to write it again. You just copy the function and change what it sends.
import smtplib
from email.message import EmailMessage
def send_digest(headlines):
body = "\n".join(f"- {title} ({url})" for title, url in headlines)
msg = EmailMessage()
msg["Subject"] = f"morning digest: {len(headlines)} headlines"
msg["From"] = "digest@example.com"
msg["To"] = "bjorn@example.com"
msg.set_content(body or "Nothing matched today.")
with smtplib.SMTP_SSL("smtp.example.com", 465) as server:
server.login("digest@example.com", "app-password-not-real")
server.send_message(msg)crontab -lMAILTO=bjorn@example.com
0 7 * * * /home/bjorn/.venv/bin/python /home/bjorn/digest.py
the NRK feed had its own small surprise
Separately from the HN crash, the NRK RSS parsing choked the first time a video-only item showed up in the feed with no article body text, just a video embed. My parser assumed every entry had a description field worth reading. That one was a much smaller fix, a plain getattr(entry, "description", "") instead of assuming the attribute exists, but it's the same lesson twice in one script: an API or a feed telling you the shape of the data is a suggestion, not a guarantee, and the edge cases show up eventually whether or not you planned for them.
the filtering itself was harder than the fetching
The actual scraping, once it stopped crashing, was the easy part. Deciding what counts as "worth my attention" is the part I keep tweaking. Keyword matching is dumb. It'll flag an article that mentions "norway" once in passing about something I don't care about at all, and it'll miss something genuinely relevant because it used a synonym instead of my exact keyword. I don't have a good fix for this. Might just be a bad problem to solve with string matching, full stop.
where it sits next to everything else
This one and the camera script are the two things I've built that actually run unattended on a schedule instead of being invoked by me typing a command. That's a different kind of programming than the CLI todo app or the PDF tools, closer to "write it once, trust it to keep working," which is a muscle I clearly hadn't built before this year, and one that apparently needs its own failure-handling discipline that a script you run by hand doesn't really punish you for skipping.
the keyword list has quietly grown
It started as three words. It's five now, and every addition came from the same pattern: I'd read something relevant on one of the sites directly, remember the digest hadn't flagged it, and go add whatever word should have caught it. "helgeland" got added after a local news story about the ferry schedule that I only found out about secondhand. It's a keyword list that only improves reactively, after it's already failed to catch something once, which isn't a great process but is apparently the only one I actually follow through on.
what I'd do differently
Fetch the fifty HN items concurrently instead of one at a time, which would also make the whole thing less painful to debug since a single slow or hanging request wouldn't stall everything behind it. concurrent.futures.ThreadPoolExecutor is sitting right there in the standard library and I still haven't gotten around to it, mostly because the thing works now and "works now" is a surprisingly strong argument against touching it again.
I'd also log a successful run, not just a failed one, so a quiet morning with genuinely zero matching headlines and a silently crashed morning with zero headlines don't look identical from the outside. Right now both produce exactly nothing in my inbox, and I can only tell them apart by going and checking the log file, which is the same mistake that caused the two-week gap in the first place, just with a smaller blast radius this time.
Anyway. It emails me headlines. Not glamorous. I read it with coffee most mornings, which was genuinely the whole goal, on the mornings it actually runs.