Weekly notes: How to find problems as a staff engineer was good read, Fabien's agents.md was great too :), Grug brained developer so much this
Still reading a lot about how teams are using agents. It’s amusing to start to see people worry more about token cost. The tool is great when it’s effectively free because it’s being susidized no question. What happens when the big ai labs start trying to turn a profit. And also it’s not clear just how much productivity these things create across the entire life of an application. Building is faster and we have some data there -hard to tell relative cost but I have to assume it’s cheaper too but the token subsidies kinda muddy the waters here- but what about ongoing maintenance, iteration, onboarding new team members, incident response, etc …
How I Find Problems to Solve as a Staff Engineer
Good reminder about how much more collaborative work gets with experience and trust. It’s not about building what I think the business needs but about listening, hearing, setting aside personal preferences and wanting to understand what others are going through day to day.
- Listen
- Talk about problems
- Wait
- Wait some more
- When I have an idea, share it
- Get feedback and listen
- Implement or write a spec that can be implemented by someone else
Book: More Everything Forever by Adam Becker
Wherein a small group of people with lots of money read a few science fiction books growing up and have some profoundly weird views about the present and the future that are being inflicted on everyone.
- AI will save us
- Our biggest problem is AI might not save us (alignment)
- Nanobots will save us
- We’ll be able to upload our brains to the internet and live forever
- We’ll be able to bring people back from the dead by feeding their writings into a chatbot
- There will be trillions of us living in space on great big space ships
- All of this really starts to happen in the next 5-10y starting with AI
- Climate change isn’t a problem because in the future we’ll be living in space
- We should live on Mars because that’ll be great
- We can burn the billions living in the present in service to the trillion trillion trillion people that will be living in space in the future
- Longtermism, effective altruism … :shivers:
- Seems like a weird cult, and
- Billionaires are really bad
Misconfiguration Spotlight: Securing the EC2 Instance Metadata Service
Server side request forgery vulnerability for applications running on ec2. IMDSv2 would be a good version of the service to migrate to.
Parse, don’t validate
I like the idea of packing more information into our data model and being thoughtful about where the responsibility for handling None values lands. Ideally it’s at the edge of the system when data is being parsed into domain objects used in the system.
Kidcameo by Jeff Woods
A website Jeff built to produce custom colouring books for families. Love the origin story and the backstage information about how the system works. Lots of ai.
How Cloudflare enforces engineering standards using AI
Cloudflare is release a bunch of documents around where they use agents in their sdlc. Here they describe a engineering codex filled with rfc style documents describing how they design and build applications. The codex is used by spec and code reviewers as well as team members who are also working in the system.
Internal agent use @ Honeycomb
There are a couple of links for this one.
- 30 to 70 PRs a Day: How We Managed to Not Wreck Our Systems, and
- AI Amplifies Your Existing Practices: Lessons from Our Shift to an AI-First Strategy
Some ideas here:
- Velocity of change increased
- Number of bugs increased (this puts a load on the team by increased number of incidents)
- Not all experiments at using ai bear fruit (team members are still backed away from automating some things)
- Ai made tiny fixes easier which delayed people taking a wider system view and refactoring
- Good practices must be in place for any of this to work (testing, monitoring, canary deployments, feature flags, fast builds)
- They have a code bot and a review bot
- Use mcp to give agents the tools and knowledge we have
- The multiplier is real but may not be the same for everyone
My Thoughts on the Bun Rust Rewrite
Can’t say I love the way Jarred interacted with the Zig community as Andrew describes it. Gross reference to Peter Thiel and some sort of ycombinator-like thing he might be running to churn out his own protoges?
Are you team A or team B?
CI pipeline logic, unless it is incredibly simple, should probably go into shell scripts for testability outside CI infrastructure which is often hard.
# good
steps:
- run: ./scripts/build.sh
- run: ./scripts/deploy.sh
# bad
steps:
- run: |
cd project
mvn package
cp target/project.war dist/
docker build Dockerfile
docker tag ...
docker push
etc.
The former is much more easily tested locally.
The Grug Brained Developer
Fantastic and weird hard learnings over a career of software development. A few of my favourites:
- complexity bad
- testing
- test first meh
- units ok but lots of churn
- e2e good but expensive to run and hard to debug
- integrations are the sweetspot
- mocks suck (they hide feedback in unfortunate ways sometimes)
- refactoring without understanding why a thing works the way it does mostly fails
- microservices suck lol
- repetition is ok sometimes
- locality of behaviour > separation of concerns (sometimes it’s better to have code all in the one place vs many abstractions and indirection to get at the lines of code that do a thing)
- logging is important
- when you branch (step into a function, major if-thens, etc.)
- stamp a request ids when you enter a new process
- dynamic log level
- per user context
- frontend is hella complicated (unnecessarily)
- apis should be designed based on usage by the application (not strictly domain objects)
How do programming languages impact token efficiency and correctness?
Less of a correlation between static vs dynamic languages and task performance in an llm. Token efficiency is measured in this paper. Python and javascript are at the top of the pack of languages tested (20-30 or so) but rust, go, and java aren’t that far behind …
What can we learn from that time Homer Simpson designed a car?
Look for the problems not solutions during user interviews. What might sound great for 1 person could be a terrible idea for others. Start by understanding how people are using a thing and where they’re having less of a good time or where there might be something missing. This process often takes time.
They Said It Would Cost $54 Million. We Said “No Thanks.”
Alberta gov’t assembles a small team of good product leaders, stakeholders, and engineers and they build a useful application faster and much cheaper than their standard procurement process would have using llms.
They had a ton of video of how the work was currently being done with an old, limited application that gpt was able to review to determine requirements and workflows.
Eventually in this process some non-technical people starting making requests for enhancements by submitting proof of concept experiments also written by llm.
All you may need is HTML
I love this so much. How much of what we do with computers is complexified more than it needs to be? Fabien’s website is delightful all around.
The Amazon tax
This doesn’t sound great. Ads on amazon compete with organic listings and what amazon already knows about the quality of a product. Enshitification. Protection money.
How I use AI in 2026 (Coding, Writing, Learning, Assistant-ing)
Coding
Hand-write (~paragraph) a CONCEPT.md — the theoretical Hacker News title, my project thesis, some scattered constraints
Pair with ultra code fable “Flesh out CONCEPT.md, what’s ambiguous, ask me questions, what are dimensions I’m not > considering, what API keys do you need…”
Pair with ultra code fable (or codex sol max) “Convert to TECH_PLAN.md, here’s how much I’m willing to spend, host on > …, here’s some API keys …”
Then I will literally just prompt “Build and verify TECH_PLAN.md” and over the next 4-48 hours I’ll let it build everything out.
and …
95%+ of the code is written in that first mega build run. I don’t think folks appreciate how much shifting left is the secret weapon against codebase slop (i.e. SlopCodeBench). Like step (4) really is binary here — there’s no pairing or even reading what the terminal agent says. If the output is wrong, I throw it completely away and add constraints to the CONCEPT.md. For many vibe coders out there, the first build prompt writes 5% of the code and I think that actually underlies most of their issues.
He learns what he can from the result without getting attached to any output …
I read the code a little bit. Often the shape (i.e. file tree) and entry points. If there’s some core algorithm, I’ll ask for a .html explainer rather than digging through the source. If I do end up digging into the code, it’s because I suspect some sort of “cheating” in the implementation.
I don’t think any of these experiments are against existing codebases.
The outcome of these projects is often an insight or the answer to the what-if question. Rarely does it make sense for me to share the code or even the app URL. Instead I typically consider the entire loop and its artifacts ephemeral and just share the insight on X or with a blog post.
No concern about any societal issues with this stuff at all.
Research / learning
See hot new research paper on X
Skim the abstract, throw the full text into Claude/Codex, “build an interactive playground artifact to explain what’s > novel here, I’m a technical person who already knows …, I’m less familiar with …”.
Play with the .html file
Ask some follow-up questions that generate an updated .html file, go to (3)
Cost
Claude Code Max 20x ($200/mo)
ChatGPT Pro 20x ($200/mo)
Google AI Pro ($20/mo) — a handy AI family plan with GSuite benefits
Modal, Railway, Netlify ($20-500+/mo) — for hosting or running experiments
How Stripe uses graph search and state machines to auto-remediate a global database fleet
Neat description of automation around autoremediation for failing / failed mongodb clusters at stripe.
What is a Harness?
An agent harness provides an environment for an ai model to operate in.
A harness includes a system prompt, tools, and a loop.
My agents.md to improve LLM-assisted code quality
This is kinda nice. I think directives like this can go into a personal claude.md or agents.md.
# FAB's AGENT.MD
- When writing something intended for human consumption, (comment, commit message, reply to prompt) use as few words as possible. Pick every word meticulously to reduce the volume to a strict minimum. Be down to the point. Less is more.
- Avoid superlatives and praise. Stop telling me I am absolutely right. Give me the cold hard truth.
- Avoid magic numbers and strings by extracting recurring or meaningful values into descriptive constants (const) or enums. Keep self-explanatory, one-off values inline to avoid clutter. If a value comes from a spec (e.g. HTTP 200 OK), use a constant regardless.
- Reduce code indentation. Avoid Arrow Anti-Pattern. Leverage early return and continue.
- Keep function names short. Less than 30 characters.
- Use enums instead of booleans for function parameters.
- Let the reader of the code breathe. Add empty lines between logical blocks of code.
- Add a small, to the point, comment to explain *what* the block does and *why*. Use examples when possible. Propose ASCII drawings to explain complete systems.
- Treat member visibility changes as a breaking design shift. Keep all fields and functions private unless external access is strictly required by the design. Prompt the user for explicit approval before changing any access modifier from private to internal or public.
- Program to levels of abstraction. Lower-level mechanics (e.g., raw hardware I/O, sector parsing, direct socket streams) must be encapsulated in a dedicated driver/abstraction layer. Expose clean, high-level APIs to the rest of the application so calling code works with domain concepts, not raw implementation details.
- Don't touch blocks of code unrelated to the feature you implement. e.g. Don't add comments to a block of code if you did not create it or modify it. As much as possible try to minimize the number of changed lines when implementing a feature.
- Strictly adhere to the layered boundary hierarchy: each layer may only communicate with its immediate neighbor directly below it. Never "punch holes" through layers (e.g., controllers or UI components must never directly call database queries, raw hardware drivers, or low-level network clients; always route through the intermediate service/abstraction layer).
- Always use {}, even on a one-line "if" statement.
When you write a commit message, follow these 7 rules:
Rule 1: Separate the subject line from the body with a single blank line.
Rule 2: Limit the subject line to 50 characters (72 is the absolute hard limit).
Rule 3: Capitalize the first letter of the subject line.
Rule 4: Do not end the subject line with a period.
Rule 5: Use the imperative mood in the subject line (e.g., "Fix bug," "Add feature,"
not "Fixed" or "Adds"). Test formula: It must complete the sentence: "If applied,
this commit will [your subject line here]".
Rule 6: Wrap the body text manually at 72 characters to prevent Git formatting issues.
Rule 7: Use the body to explain what and why vs. how. Assume the code explains the how;
the message must explain the context and reasoning.
- If the prompt indicates that a bug is being fixed, don't write the fix right away. First write the test. Observe it failing. Then write the fix. And observe the test passing.
Getting the Most Out of 1:1s
Good overall article about 1:1s. My favourite points:
- Weekly to start and might become less frequent when a new team member has settled
- Place to provide feedback to a team member
- Place for them to talk about problems they see (might need a bit of time to build trust)
- Good question : What’s a [design/technical] choice you made this week that we can talk through?
How to Build Your Infrastructure Monitoring in 2026
Planning for monitoring a system:
- Understand critical business flows
- Know what good looks like for them (Can define SLIs / SLOs / error budgets)
- Build / buy a stack for metrics, traces, logs
- His preferred
- VictoriaMetrics for metrics (vmagent for metrics collection)
- Loki for logs
- Jaeger for traces
- OpenTelementry for log, trace collection
- His preferred
- Correlation ids for relating log events
- Rate, error, duration dashboarding initially
- Alert on symptoms not causes (High cpu isn’t a good indicator of trouble - fewer logins might be)
snakes.run: rendering 100M pixels a second over ssh
Delightful implementation of snakes. This takes me back to my early days of computing and games written in basic. (Nibbles and Gorilla specifically :)) With an added element of multiplayer over ssh! Amazing. This is a writeup of some of the work that went into making it.
Primary concerns were bandwidth initially to manage unbounded cost and then performance. Lots of profiling later improvements were made by moving away from generalized functions to snakes.run specific ones.
Garbage collectors must always be respected. :)
Legibility of Effort
It’s hard to tell whether a website has been made with love or by clankers. I am very much less interested in the latter. Or differently interested? Maybe that’s not completely fair but there is something special about art created by people who have put in the effort and this was how everything used to be until recently.
Agentic Engineering at Zalando: a snapshot
Interesting evolution of agent use over 2 years. Started with an ai api gateway that was monitored to see how people were using llms. They had to figure out how to manage increased complexity in prs. (It’s acknowledged but I’m not sure what if any policy was changed to account for it. They do have an auto-approve process step for “less complex” prs.)
There’s also detail about how to share learnings across the team and allowing freedom to explore. It sounds like they have a global libray of skills.
Control and complexity: tension in systems design
Two different ways of thinking about systems:
- decomposable, understandable in parts, whose can be understanding from parts
- systems are complex with emergent behaviour
He frames activities with different words depending on the point of view:
- eg training and education
- DECOMPOSE / ANALYZE: What is known about a system can be taught, most things about a system are known and at least knowable
- COMPLEX / EMERGENT: Create an environment where we can experiment, and explore, and guide
It seems like both of these world views are useful.