Backups that actually work
Backups are now one package per backup, with the database dump, blob storage archive and volume, downloadable as a single checksummed .tar.gz. We also caught and fixed a bug that had been breaking our own MySQL dumps.
Today we shipped backups for PicoPaaS apps, and just as importantly, we proved they work. This post covers what a backup is now, how you get one out, a bug in our own backups that we caught along the way, and what is still missing.
One backup, one package
Each backup is a single backup package. It holds whatever the app has: the database dump, the blob storage archive and the persistent volume. The parts are not taken at the same instant, but they are grouped and handled as one backup. One download gives you a single .tar.gz containing:
database.sql.gz(MySQL) ordatabase.pgdump(PostgreSQL)blob-storage.tar.gzvolume.tar.gzMANIFEST.json, with a sha256 for each part
One delete removes the whole package. Every backup is encrypted with a per-app password.
Opt-in, and what we keep
Scheduled backups are opt-in per app, daily or weekly, set on the app's Settings page. "Back up now" works either way, once per hour per app. Retention keeps the latest daily, weekly and monthly backup. That is not an archive, so if you want something long term, download it.
Downloading
The package is prepared in the background, so you can leave the page and come back within an hour. Each click gets a fresh 10-minute link. The prepared file is deleted after an hour, and there is a limit of 3 downloads per hour per app. From the CLI:
picopaas backup list|run|download|enable|disableThe bug we caught
Our own MySQL dumps had been failing since a "timeouts" fix the previous night. That fix passed --connect-timeout to mariadb-dump, which rejects it as an unknown variable. Our unit tests stubbed the database tools, so they never noticed. The dump now runs under timeout instead.
To keep this from happening again, every build of the backup image now runs a smoke test against real MySQL 8.4 and PostgreSQL 17, with TLS on. It covers backup, restore, deleting a snapshot, a volume backup, and building a package whose contents and checksums are then compared. The image is published only if it passes. The smoke test also caught a bug in itself on its first CI run: a directory-permission difference between Linux and Docker Desktop.
Checked in production
- We downloaded a backup of the MySQL database behind this blog: 75 tables, a complete dump.
- We downloaded a PostgreSQL (n8n) backup and restored it into a fresh PostgreSQL 17. All 142 tables loaded with no errors.
- We backed up a throwaway WordPress with a 1 GiB volume as one package of database plus volume. The download was about 34 MB, both checksums matched, and the volume archive held
wp-config.php,wp-contentandwp-includes. Then we deleted the package as one unit.
Security
Backups use a storage key that can only reach the backup bucket. On every run, the deploy proves the key cannot read any other bucket, and volume backups switch on only if that proof passes. Volume backups run as a short-lived job in the app's own namespace, the only place the volume can be mounted. The volume is read-only, egress is limited to the backup storage, and the per-run secret is removed afterwards.
We did two security reviews. Among other things they found and fixed download file names built from app names, object names containing .. in a bucket, a double-click race (now an app-level database lock) and leftover per-run secrets. Everything is audited: creates (manual and scheduled), downloads, each download link issued, deletes, retention pruning and purges.
Not done yet
You cannot restore a backup through the UI yet. A restore drill on a throwaway app is next, and we will write that up when it is done.