Kaisen learns to check its own work

Kaisen (in preview) can now edit and search files, check its own endpoints, continue past 40 turns and restore earlier files, and we are about to run a seven-brief eval to see whether that lets cheap models build bigger apps.

Share

Kaisen is in preview, and we just shipped the biggest change to how it works since it launched. The goal: let it build apps bigger than a single page without falling over, and let it check its own work before it says it is done. Below is what changed, and the test we are about to run to find out whether it worked.

Why bigger apps were hard

  • To change a file, the builder rewrote the whole file. On a larger app that hits the model's output limit.
  • It had no way to search the code, so it re-read files to find things.
  • The only check was a health check. A broken feature behind a working home page looked fine.
  • Mistakes showed up only after a deploy, which takes one to four minutes.
  • Long builds stopped at 40 turns.
  • There was no undo.

The fixes borrow ideas from Anthropic's Effective harnesses for long-running agents and the app.build paper. Thanks to both for doing the thinking in public.

What shipped (live now)

  • edit_file and search_files. The builder changes one exact piece of text at a time and can search across files, so edits to a bigger app cost fewer tokens.
  • Instant warnings. Each file write is checked for Python syntax errors, invalid JSON and kaisen.json mistakes, with no deploy needed.
  • http_request. Once a preview is up, the builder can send requests to that build's own preview and check each feature's real endpoints before finishing. It cannot reach any other site. The reviewer gets a read-only GET.
  • One feature at a time. PLAN.md is worked through feature by feature.
  • Auto-continue. Instead of stopping at 40 turns, a run chains up to 4 steps, each in a fresh context seeded from PLAN.md. Long resumes start fresh too.
  • Restore points. A snapshot is saved at each run start, the last 5 are kept, restoring is undoable, and you get a diff per run and per restore point. For full history, send the output to GitHub.
  • Runtime image re-pinned, so generated tests now report their results on the build page.

Details are in the guide: what the builder can check, restore points and Kaisen limits.

Security

Both batches were reviewed before shipping. Two MEDIUM findings were fixed: a regex search that could stall the shared worker, and a diff that could burn CPU. Several LOWs were fixed too (gzip bombs, slow responses, credential echo, a decompression bomb). No HIGH or CRITICAL findings. The endpoint tool can only reach the build's own preview.

The test we are about to run

Shipping tools is not the same as them working. So we are running an eval: a script feeds fixed briefs through the public API, on a test account, in a fresh project (not the demo project). It is never run in CI. There are seven briefs: the five starters (Fancy Docs, Team standup board, Link shortener, Habit tracker, Recipe scaler) plus two harder ones. One is a multi-user kanban board with MySQL and a JSON API. The other is a notes app with file uploads to a bucket, and search.

Each brief has endpoint checks the script runs against the live preview after the build. For the kanban board: /, /health, and an unauthenticated /api/cards must return 401 or 403. For notes: create a note, then search must find it.

Batch 1 runs with the reviewer on. We report the cost before running the rest.

BriefModel
Link shortenerClaude Haiku 5.5
Kanban boardClaude Haiku 5.5
Notes with uploadsClaude Haiku 5.5
Kanban board (same brief)Claude Sonnet 5.5

The Sonnet run is there for a direct comparison. Haiku 5.5 is $0.10 / $0.50 per million input / output tokens; Sonnet 5.5 is $2 / $10, which is 20x. Our earlier five Haiku demos cost $0.14 in total.

For every brief we record: finished or not; runs, steps and context resets; cost; wall time and time to first working deploy; test result; reviewer verdict; endpoint checks passed; tool usage (edits versus rewrites, searches, endpoint checks, failed deploys, write warnings); stops (repeat detector, output limit, turn limit); PLAN.md items ticked versus unticked; code size; and whether tests, a README and a non-root Dockerfile are present.

The questions we want answered:

  • Does Haiku handle multi-feature apps now?
  • Is Sonnet worth 20x?
  • Do the new tools actually get used?
  • Do builders verify features, or just the home page?

The results, and what we change because of them, will be a follow-up post. Kaisen is in preview, so expect rough edges.