Devlog 2026-10-23: admin tools, a pentest pass and a database scaling review

Today we shipped pod events and audit log downloads, admin promote and demote with a recovery CLI, fixes from our own pentest, and the first four database scaling fixes. The live end-to-end run passed 19 tests with none failing.

Share

A busy day on PicoPaaS. Most of it was admin tooling, then a security pass on that tooling, then a look at how the platform database will hold up as we grow. Here is what landed.

Docs split in two

The admin guide used to mix generic platform administration with notes about our own instance. We split it. ADMIN_GUIDE is now for anyone running PicoPaaS. The specifics for this deployment (infrastructure, CI/CD, runbooks) live in the repo under docs/deployment/.

UI tweaks

  • Nav links now have a stronger hover background, so the hovered item is easy to see.
  • The apps list has a "View app health" icon that jumps straight to the app's health page.
  • The pod picker marks which pod is current, and wraps properly on narrow screens.

Downloads for debugging and audits

You can now download a pod's Kubernetes events as a text file (or JSON) from the pod picker, which is handy when a pod will not start and the logs are empty. Admins can also export the audit log as CSV or JSON. The export is safe to open in a spreadsheet: cells that start with a formula character get neutralised. Exports are themselves audited.

Cluster health also shows how many apps use MySQL and how many use PostgreSQL.

Promote, demote and break-glass

Platform admins can now promote and demote other users from the Admin page. Demoting, suspending or deleting the last active admin is refused. If you ever do lock yourself out, there is a recovery CLI that runs where the app runs:

python -m app.manage list-admins
python -m app.manage grant-admin <email>

We pentested ourselves

We pointed a security review at the new features and found three things, all fixed the same day:

  • Cross-site request forgery from sibling subdomains. Tenant apps live on subdomains of the same site as the dashboard, so browsers treat them as same-site. Cookie-authenticated writes now go through an Origin and Sec-Fetch-Site guard.
  • A last-admin race. Two concurrent demotions could have removed every admin. The check now runs under a row lock.
  • Suspended admins. The admin gate now requires an active account, not just the admin flag.

The UI pass also caught a missing RBAC permission: listing events cluster-wide was not allowed, so the pod events download failed on the live cluster. That is fixed too. The pentest write-up is in the repo.

Tests

A coverage pass added 36 tests for the admin features: edge cases for the current-pod marker, event download errors, audit export caps and formula prefixes, last-admin rules and the manage CLI. It found no app bugs, and overall coverage stays at 92 percent.

Database scaling review

We also reviewed how the platform database behaves as the number of apps grows (the write-up is docs/research/db-scaling.md, 14 findings). Four quick fixes shipped from it:

  • Pool sizing. Connection pool size, overflow, timeout and recycle are now environment knobs with conservative defaults.
  • Fewer ALTER USER calls. The per-app database connection limit used to be reapplied on every 15-second reconcile sweep. It is now applied only when it changes.
  • Eager loading. The reconciler and the app list load related rows in batches instead of one query per app.
  • New indexes on the columns those queries filter by.

We deliberately deferred the audit retention job. It is not urgent at current volumes.

Does it still work?

After deploying, we ran the live end-to-end suite against production, including the slow quota tests: 19 passed, 1 skipped (it needs a big test image), 0 failed, and no leftover test apps.