Devlog: Kaisen hang watchdog, 95% coverage, app research, and an AWS plan
Tonight: a Kaisen hang watchdog with an Interrupt and retry button, test coverage up to 95.25%, research on the next app templates, and a not-yet-approved plan for an AWS test environment.
A quiet-looking night with four threads of work. Kaisen got a hang watchdog, the test suite got a lot more coverage, we researched which apps to template next, and we wrote down the plan for testing on AWS. Some of this is live and some is only planned, so each section says which.
Kaisen no longer hangs forever
Agents can hang: a model call or a tool that never returns. We now bound everything. Every model call has a hard wall of 540 seconds and a stream idle timeout of 300 seconds, and tools are capped at 420 seconds. If a run makes no progress for 10 minutes, it is interrupted and resumed from its last saved state. After 2 automatic retries it stops with "Hung N times; Resume to retry".
- Late answers from abandoned attempts are fenced off and discarded, so a zombie call cannot overwrite the retry.
- The worker heartbeat only beats while the main loop makes progress, so a wedged worker gets restarted by Kubernetes.
- Members get an "Interrupt and retry" button. It is rate limited to 3 per build per hour, because an abandoned model call is still billed but not metered.
The honest bit: our first version crashed Python in the test suite. A background heartbeat thread queried SQLite after test teardown. The heartbeat is now database-free. One caveat: the backend and tests are done, but we have not yet checked the new button in a browser.
Coverage
Overall coverage went from 92.77% to 95.25% across 2602 tests, and the CI floor moved from 92 to 93. The Garage object store went from 23% to 96%, the Kaisen API from 88% to 98%, and the Kaisen kit repo service from 90% to 93%.
Which apps next? Research only
We surveyed self-hostable apps as template candidates. The top five are Vikunja, Planka, Umami, ntfy, and IT-Tools/Excalidraw. These are candidates, not shipped templates. Some surprises: Uptime Kuma's full image is 617 MB against the 512 MB cap, so only the 2-slim variant fits, and Linkding only works on its plain alpine image. A few projects are archived or moved (Pingvin Share, PocketBase's repo, File Browser). PostgreSQL is the more useful engine for new templates.
AWS: a plan, nothing built
We wrote an account setup guide and an infra request. The request is proposed, not approved, and no AWS resources exist yet. The throwaway test environment would cost about $0.27 per hour and about $0.51 a month idle. Budget data lags by hours, so the real caps are a low vCPU quota and a nightly TTL teardown. CI would use GitHub OIDC roles behind an environment that requires a reviewer.