Same model, 5x the bill, depending on the harness. Tech & AI Weekly by Eli, September 18 2026
Hi!
Another week, another round of coding-agent headlines. What’s underneath them keeps saying the same thing I said two weeks ago: the model was never the hard part. This week the receipts piled up: the harness you wrap around a model swings its cost 5x, and the best agent alive still trips the moment it meets code it hasn’t seen.
🤖 AI & GenAI
Same model, same score, up to 5x the cost, depending on the harness · Berkeley’s Sky Lab tested 21 model-harness pairs (Claude Code, Codex CLI, and a bare open-source harness called Pi). The harness barely moves success rate but swings cost up to 5x: Fable 5 passes ~97% in all three, yet Claude Code costs double what Pi does, partly because its first model call carries 10x the context. They call it a “harness tax”, and most of us pay it by taking the default. Read more
The best coding agent still fails 60% of the time, on code it hasn’t memorized · Real-SWE drops agents into private company codebases instead of public repos, and the top scorer (Claude Fable 5.1) hit just 38.8%. The reasoning is fine; what’s missing is the context. On familiar public benchmarks the model half-remembers the answer. Drop it into an unseen repo where the fix touches 11 files it has to find first, and a huge context window doesn’t save you: storing everything isn’t the same as retrieving the right thing. Read more
GitHub rewrote the Copilot runtime in Rust, using Copilot · 800,000 lines of production Rust across 128 PRs, agents wrote most of it, shipped incrementally instead of one terrifying cutover. The part worth keeping isn’t “AI wrote code”, it’s that one engineer ran the control loop on a migration a whole team usually never gets to greenlight. Read more
☁️ Cloud & DevOps
AgentCore can now rewrite your agent’s system prompt from real traces · It reads production traces, finds the failure patterns, and proposes a revised prompt scored against an evaluator you pick. Prompt tuning stops being vibes and starts being an experiment. Read more
Bedrock prompt caching, for cost and latency · A straightforward win if your agent resends the same big context on every call. Worth knowing where it helps and (spoiler, see below) where it quietly doesn’t. Read more
✍️ From me this week
-
Semantic Caching for AI Agents in Production · Prompt caching only reaches one of the five layers an agent can cache. I built the same semantic cache on two stores (DynamoDB vector search and ElastiCache for Valkey), measured what actually differs, and found the trap that breaks most of them: “flights on Sep 15” and “flights on Dec 15” score 97% alike, so closeness alone will happily hand you the wrong date. The fix is a guard that checks the numbers, not the wording.
-
How to Stop AI Agent Memory Poisoning · One malicious message your agent stores as a fact, then acts on forever. I measured the blast radius: one poisoned fact skews 1 of 4 answers in key-value memory but hijacks 4 of 4 in a graph, because it wires a conflicting decision edge onto the traveler. The defense is a write-gate inside the agent’s harness, so every entry point is covered, not just one app.
📅 Where to find me
-
Graphs for Engineering (Neo4j + AWS Builder Loft, San Francisco, Sep 22) · I’m giving “The Three Hidden Graphs in Every AI Agent”: the context, execution, and provenance graphs hiding in every agent, and what you get when you make them explicit. Four talks and food, in person. Details
-
WeAreDevelopers World Congress, North America (San Jose, Sep 23-25) · Workshop: “Reduce AI Agent Costs and Mistakes with Semantic Tool Selection”, building two versions of the same travel agent side by side so you can watch the accuracy and token savings for yourself. Come find me if you’re there.
I hope you enjoy this as much as I enjoy putting it together every week. It’s my little weekly de-stress moment. Let’s stay informed together and learn something new along the way.
See you next Friday,
Eli
Dev.to · LinkedIn · GitHub · Twitter/X · Instagram · YouTube
¿Qué te pareció esta edición? Solo responde a este email.