Kaisen eval, round 3: the fixes land
Round 3 of the Kaisen preview eval: the fixes held, a new Small CRM failure was found and fixed the same day, and Haiku now matches Sonnet on every brief at a fraction of the cost.
Kaisen is in preview, and round 3 of the eval is where the fixes from round 2 got their test. Short version: they held, and one new failure turned up and got fixed the same day. These are signals from a small sample, not benchmarks.
What we ran
The new Kaisen runtime image is live, so static sites now run their tests ("test": "node --test") and the runtime logs which bundle it serves. We re-ran the briefs the fixes touched on Haiku, $0.20 in total.
- Notes with uploads: 4/4 checks, tests pass, approved. First time on Haiku.
- Link shortener v2 (JSON API, 6-step check including a 400 for a bad URL and a 409 for a taken code): 6/6, tests pass, approved.
- Recipe scaler (static): the reviewer flagged that tests never ran. The builder answered each finding: "fixed" twice and one reasoned "disagree" (it can't run a browser, and the logic is pure JS). The static tests then ran on the new image and passed 13 of 13.
- Small CRM: failed first. Haiku tried to write the whole app in one reply, hit the 32,000-token output limit and lost it, twice. 0 files.
The CRM fix
A cut-off reply now gets one retry, with a note to write one or two files per reply and long files in parts. A second cut-off still stops. Only Kaisen can write that note, so app output can't fake it. The rerun finished: 12/12 checks, tests pass, 29 steps, $0.06. On Sonnet two days ago the same brief cost $0.63.
Around the edges
Production deploys now wait for green CI on the same commit, and only main's newest commit deploys, so a late-finishing older commit can't roll production back. A push reaches production about 20 minutes later. CI also gained ruff (93 findings, all dead code or style, no real bugs), shellcheck and actionlint. The first lint run failed because actionlint also checked the inline shell in four workflows; that check is off for now. Dependency bumps merged too, including the Anthropic SDK from 1.17 to 2.3, checked against the Kaisen tests and a few real calls.
Still to do
- Auto-continue after 40 turns has never triggered in a real build (the biggest was 29). A deliberately big brief is next.
- Tests run without database credentials by design, so database tests skip. Options: a throwaway test database at boot, or document it.
- One run per brief and model is thin. Plan: dated summaries and two runs per brief.
- Haiku now matches Sonnet on every brief at a fraction of the cost, so the default model gets decided on data.
- Re-enable shellcheck for scripts inside workflows.
Up next: project suspension, for suspended users and for admins handling abuse in shared projects.