Kaisen eval, round 2: what we fixed and what's next
Round 2 of the Kaisen preview eval: Sonnet finished 8/8 briefs for $1.79, we fixed the deploy check, static-site tests and reviewer follow-through, and here is what is still open.
Kaisen is in preview, and last time we ran it against eight starter briefs and wrote down what broke. Tonight we fixed those things and ran it again. Small sample, one run per brief and model, so treat all of this as signal, not benchmark.
Round 2 results
Same eight briefs on Sonnet and Haiku, in a separate eval project, with the stricter reviewer running on Haiku and web access off. Plus one web brief.
- Sonnet: 8/8 done, 29/29 checks, $1.79, 28 minutes. Tests pass on all six server apps; the two static sites had none.
- Haiku: 6 of 8 done, 13/13 checks, $0.18. Then our eval script crashed on one network timeout. It now retries, and a failure only fails that brief.
- Same six briefs: Sonnet cost 5.5x Haiku. It took far fewer steps and was faster, but wrote thinner tests.
- Web brief: a weather lookup on Haiku, limited to open-meteo.com. Two searches, two docs pages, then an app calling Open-Meteo live. Paris returned real data, an unknown city got a 404. $0.045.
Round 1's fixes held: every server app has a captured test result, logged-in checks work, every build finished with PLAN.md ticked, and the reviewer asked for changes on 5 of 8 Sonnet builds, mostly with real findings.
What we found and fixed
- Deploy check. It probed only
/and accepted only 2xx, so a login-gated CRM that correctly redirected to/loginlooked broken and Sonnet redeployed 8 times. It now probes the declared health path and counts a 3xx as answering. - Rollouts. The check could probe the old version. An app now counts as deploying until every pod runs the new one. This applies to every PicoPaaS app: a release whose new pod never gets ready fails after the deploy timeout and rolls back.
- Static sites got conflicting advice on tests. The runtime now runs their tests (
"test": "node --test"). Side lesson:node --test tests/fails on Node 22+, and our smoke test caught it before release. - Reviewer fix rounds went unchecked; 3 of 7 builders finished again without changing anything. The builder must now answer each finding with "fixed" or "disagree", and the build page shows the answers.
- Security: web tools are off for any build with secrets, even with an allow-list, since a prompt-injected page could trick the builder into writing code that leaks them.
All of it was security-reviewed (findings K36-K44: one medium, accepted on purpose, plus a few lows, all fixed). Production deploys also now wait for green CI on the same commit, main is protected against force-pushes, and CI has a 92% coverage floor.
Still to do
- The new runtime image isn't live yet; it publishes, then gets pinned.
- Haiku reruns of the briefs these fixes touch (Notes, Small CRM, Link shortener, static sites).
- Auto-continue has never triggered in a real build. Small CRM on Haiku should be its first test.
- The eval itself: one run per brief, and 4 of 8 briefs only check the home page loads. Next: two runs per brief, bigger briefs, run before any prompt change.
- Default model: Haiku matched Sonnet on these briefs at a fifth of the cost. We'll decide on data.
Total spend tonight: about $2.