Kaisen eval, round 1: what we found
We ran four Kaisen builds (in preview) for about $0.20 total, found six things to fix, and shipped every fix before the full eight-brief run.
Last time we described the eval plan for Kaisen, which is in preview. Before spending real money on all eight briefs, we ran four builds to see what it costs and what breaks. Here is what we found, and what we already fixed.
The setup
A separate eval project, the test user's own API key, effort medium, the reviewer on (Haiku), web access off. Three briefs on Haiku and one on Sonnet.
| Brief | Model | Cost | Turns | Time | Checks | Review |
|---|---|---|---|---|---|---|
| Link shortener | Haiku | $0.017 | 13 | 2.9 min | 1/1 | approve |
| Kanban board | Haiku | $0.029 | 19 | 4.1 min | 3/3 | changes requested |
| Notes with uploads | Haiku | $0.026 | 12 | 3.4 min | 4/4 | approve |
| Kanban board | Sonnet | $0.128 | 5 | 1.9 min | 3/3 | approve |
Total: about $0.20. Sonnet cost roughly 4.4x Haiku on the same brief, not 20x, because it needed far fewer turns.
What went well
All four builds got the manifest, database, bucket, non-root user, README and tests right. There were 0 write warnings and 0 failed deploys. The new tools got used: edit_file 10 times, search once, http_request 20 times.
What we found, and fixed
- The results of the tests the builder wrote weren't captured. The worker now waits for them after a healthy preview deploy.
- http_request kept no cookies, so nothing behind a login was checked. It now has a per-run cookie jar, and results show cookie names only.
- Three of four builds finished with PLAN.md unticked. The first finish with open items is now refused once.
- Sonnet's work was thin (a 603-byte test file, one deploy, two HTTP calls) and the Haiku reviewer approved it. The prompts now ask for tests per feature, and the reviewer sees a redacted summary of the builder's HTTP calls.
- No build came near 40 turns, so auto-continue wasn't exercised. We added a bigger Small CRM brief, and Kanban now requires signup and login.
- The eval read results before the final redeploy settled. It now waits.
A caveat
Four builds is a tiny sample. Treat these as signals, not benchmarks.
The full eight-brief run is under way; a later post will cover it.