Kaisen eval, round 1: what we found

We ran four Kaisen builds (in preview) for about $0.20 total, found six things to fix, and shipped every fix before the full eight-brief run.

Share

Last time we described the eval plan for Kaisen, which is in preview. Before spending real money on all eight briefs, we ran four builds to see what it costs and what breaks. Here is what we found, and what we already fixed.

The setup

A separate eval project, the test user's own API key, effort medium, the reviewer on (Haiku), web access off. Three briefs on Haiku and one on Sonnet.

BriefModelCostTurnsTimeChecksReview
Link shortenerHaiku$0.017132.9 min1/1approve
Kanban boardHaiku$0.029194.1 min3/3changes requested
Notes with uploadsHaiku$0.026123.4 min4/4approve
Kanban boardSonnet$0.12851.9 min3/3approve

Total: about $0.20. Sonnet cost roughly 4.4x Haiku on the same brief, not 20x, because it needed far fewer turns.

What went well

All four builds got the manifest, database, bucket, non-root user, README and tests right. There were 0 write warnings and 0 failed deploys. The new tools got used: edit_file 10 times, search once, http_request 20 times.

What we found, and fixed

  1. The results of the tests the builder wrote weren't captured. The worker now waits for them after a healthy preview deploy.
  2. http_request kept no cookies, so nothing behind a login was checked. It now has a per-run cookie jar, and results show cookie names only.
  3. Three of four builds finished with PLAN.md unticked. The first finish with open items is now refused once.
  4. Sonnet's work was thin (a 603-byte test file, one deploy, two HTTP calls) and the Haiku reviewer approved it. The prompts now ask for tests per feature, and the reviewer sees a redacted summary of the builder's HTTP calls.
  5. No build came near 40 turns, so auto-continue wasn't exercised. We added a bigger Small CRM brief, and Kanban now requires signup and login.
  6. The eval read results before the final redeploy settled. It now waits.

A caveat

Four builds is a tiny sample. Treat these as signals, not benchmarks.

The full eight-brief run is under way; a later post will cover it.