Backup and restore
A LodeBase server keeps its state in three places. A backup needs all three.
| What | Where | How it's backed up |
|---|---|---|
| Postgres | repos, changesets, users, orgs, locks, merge requests, pipeline runs, encrypted secrets | pg_dump, every run |
| Object store | file contents, folders, artifacts (LODEBASE_STORAGE) | copied after the dump; append-only |
| Secrets master key | LODEBASE_SECRETS_KEY (and the rest of your .env) | by you, once: password manager or secrets store |
Without the master key the restored server works, but pipeline secrets can't be decrypted: you'd set them again. Keep it somewhere other than the backups.
Why the order matters
Objects are named by their hash and never change, and a row in Postgres is written only after its object is stored. So a dump taken first, followed by a copy of the object store, always has every object it references, as long as nothing is deleted from the store in between. The backup script pauses garbage collection's deletes for that window (the pause expires on its own after PAUSE_HOURS, default 12, if the script dies). The server keeps running and accepting submits throughout; changes after the dump started just aren't in this backup.
Back up
deploy/backup/lodebase-backup.sh does all of it:
sh
DATABASE_URL=postgres://lodebase:PASSWORD@localhost/lodebase \
LODEBASE_STORAGE=file:///var/lib/lodebase/objects \
BACKUP_DIR=/backups/lodebase \
deploy/backup/lodebase-backup.shIt writes:
/backups/lodebase/
db/lodebase-20261114T030000Z.dump pg_dump, custom format (the newest KEEP_DUMPS=14 are kept)
db/lodebase-20261114T030000Z.json stamp, schema version, sizes
objects/ the object store; only new objects are copied each timeEvery kept dump restores against objects/, because nothing is ever removed from it. It only grows: to reclaim space, start a new backup directory now and then (say monthly) and delete the old one once the new one has a few dumps in it.
With Docker Compose
Postgres runs in its container and the objects live in a Docker volume:
sh
cd /srv/lodebase
PG="docker compose -f deploy/docker-compose.yml exec -T postgres" \
DATABASE_URL=postgres://lodebase:PASSWORD@localhost/lodebase \
LODEBASE_STORAGE=file://$(docker volume inspect -f '{{.Mountpoint}}' deploy_objects) \
BACKUP_DIR=/backups/lodebase \
sudo -E deploy/backup/lodebase-backup.sh(PG runs pg_dump and psql inside the container, so the client version always matches the server. The volume name is <compose project>_objects; docker volume ls shows it.)
With S3, Garage or MinIO
The object copy uses rclone. Create a remote once, then point the script at it:
sh
rclone config create lodebase-s3 s3 provider=Other endpoint=https://s3.example.com \
access_key_id=... secret_access_key=...
LODEBASE_STORAGE=s3://lodebase RCLONE_REMOTE=lodebase-s3 ... deploy/backup/lodebase-backup.shTo keep the object copy in another bucket instead of on disk, add OBJECTS_DEST=backup-remote:lodebase-objects. Bucket versioning or replication on the storage side is a good extra, not a replacement: it doesn't give you a database dump that matches.
Schedule it
Nightly with a systemd timer (or cron: 30 3 * * * root /srv/lodebase/deploy/backup/lodebase-backup.sh):
ini
# /etc/systemd/system/lodebase-backup.service
[Service]
Type=oneshot
EnvironmentFile=/etc/lodebase/backup.env # DATABASE_URL, LODEBASE_STORAGE, BACKUP_DIR, PG
ExecStart=/srv/lodebase/deploy/backup/lodebase-backup.sh
# /etc/systemd/system/lodebase-backup.timer
[Timer]
OnCalendar=*-*-* 03:30
Persistent=true
[Install]
WantedBy=timers.targetOff site
BACKUP_DIR on the same disk as the server protects against mistakes, not against losing the machine. Send it somewhere else too. restic works well because the dumps and objects deduplicate:
sh
restic -r sftp:backup@nas:/lodebase backup /backups/lodebase
restic -r sftp:backup@nas:/lodebase forget --keep-daily 14 --keep-monthly 6 --pruneRestore
Stop the LodeBase server (
docker compose stop lodebase).Create an empty database. On Compose, a fresh Postgres volume:
docker compose down,docker volume rm deploy_pgdata,docker compose up -d postgres. Otherwisecreatedb lodebase.Run the restore (the newest dump, or name one):
shDATABASE_URL=postgres://lodebase:PASSWORD@localhost/lodebase \ LODEBASE_STORAGE=file:///var/lib/lodebase/objects \ deploy/backup/lodebase-restore.sh /backups/lodebase [db/lodebase-20261114T030000Z.dump]It refuses a database that already has tables. It restores Postgres, copies the objects back (to the S3 bucket for
s3://), and clears the backup's own pause.Start the server with the same
LODEBASE_SECRETS_KEY. It runs any newer migrations itself, so a dump from an older version restores into a newer server.Check it:
lode admin verifywalks every repo and confirms every object it references is in the object store. It exits non-zero and lists what's missing if not.
After a restore, clients keep working: their workspaces are at changesets the server has (unless you restored a dump older than their last sync; then lode sync brings them back in line and submits of theirs that aren't in the dump need submitting again).
Practise it
A backup you haven't restored is a hope. scripts/backup-test.sh does the whole cycle on every test run: it backs up a live server with history, an org, a shelf, a secret and pipeline artifacts, deletes the database and the objects, restores into a new database, and checks every file, the history, the shelf, the artifact zip and that a new pipeline run can still read the secret. Do the same on a spare machine with a real backup every few months.