DB-VPS backup tier

Current layout

DB-VPS has a dedicated mirrored backup tier on two 12 TB HDDs:

Layer	Value
Disks	`/dev/sdb` `ST12000NM000J-2T` serial `ZRT0E14W`; `/dev/sdc` `ST12000NM000J-2T` serial `ZRT0E14H`
RAID	Linux `mdadm` RAID1, `/dev/md/backup12t` (`/dev/md127` at runtime)
RAID UUID	`5b951241:335d55e2:31e43f2c:1b6222a0`
Filesystem	ext4, label `BACKUP_RAID`, UUID `e63b9568-f717-47cb-a9b4-4c0b0b1be4fc`
Mount	`/backup-raid` via `/etc/fstab`
Backup tool	BorgBackup, unencrypted local repositories under `/backup-raid/borg`
Monitoring	`mdmonitor.service` and `smartmontools.service`

The old 1 TB /backup disk is still mounted separately and remains the live storage for Mailcow and container registries. Do not repoint /backup to /backup-raid without a separate migration plan because Docker bind mounts currently reference directories below /backup.

Daily backup framework

The generic runner is:

/usr/local/sbin/tracium-backup-run /etc/tracium-backup/sources.d/<source>.conf

Each source config defines:

SOURCE_NAME
BORG_REPO
PATHS
EXCLUDES
optional PRE_HOOK and POST_HOOK
retention values: RETENTION_DAILY, RETENTION_WEEKLY, RETENTION_MONTHLY, RETENTION_YEARLY

Systemd units:

systemctl list-timers --all 'tracium-backup@*.timer'
systemctl status 'tracium-backup@host-ssd.timer'
systemctl status 'tracium-backup@host-ssd.service'
journalctl -u 'tracium-backup@host-ssd.service' -n 200 --no-pager

Active sources:

/etc/tracium-backup/sources.d/host-ssd.conf
/etc/tracium-backup/sources.d/tracium-postgres.conf

This source backs up SSD-host configuration and deploy state:

/etc
/root
/home
/srv
/opt
/usr/local
/var/spool/cron

It intentionally does not copy active database files or Docker runtime storage. Databases must be backed up through a dedicated source with a dump/quiesce hook.

Tracium PostgreSQL source

The tracium-postgres source creates a logical PostgreSQL backup from the tracium-postgres-1 container:

config: /etc/tracium-backup/sources.d/tracium-postgres.conf
pre-hook: /etc/tracium-backup/hooks.d/tracium-postgres-dump
post-hook: /etc/tracium-backup/hooks.d/tracium-postgres-cleanup
repository: /backup-raid/borg/tracium-postgres
timer: tracium-backup@tracium-postgres.timer
schedule: daily at 00:30 UTC plus up to 20m randomized delay (override in tracium-backup@tracium-postgres.timer.d/schedule.conf; staggered ~4h before host-ssd’s 04:30 UTC so the two backups do not contend on I/O)
retention: 7 daily, 4 weekly, 6 monthly, 1 yearly

Default mode is a full custom-format pg_dump plus pg_dumpall --globals-only. The dump is written into /backup-raid/staging/tracium-postgres and then archived by Borg. The post-hook removes the staging directory after the archive is created.

As of 2026-07-12, the tracium database is about 297 GB. The first full run was initially scheduled for 12:30 UTC (midday) so it would not compete with the initial 12 TB RAID resync. That accommodation was reverted to 00:30 UTC (night) on 2026-07-14: the ~2.5h daytime pg_dump saturated I/O on the busy DB (assignment-worker ticks 16-56s → 86-102s) and held ACCESS SHARE on every table, which blocked a DROP INDEX CONCURRENTLY migration and hung an api-server boot-goose → failed deploy. Rule: run heavy DB dumps only in the night window, away from ingestion load and deploy migrations.

A schema-only smoke archive was created successfully:

sudo env TRACIUM_POSTGRES_DUMP_MODE=schema \
  /usr/local/sbin/tracium-backup-run \
  /etc/tracium-backup/sources.d/tracium-postgres.conf

Smoke archive:

tracium-postgres-20260712T182004Z

Run the full PostgreSQL backup manually only when the RAID is not doing heavy rebuild work:

sudo systemctl start 'tracium-backup@tracium-postgres.service'
journalctl -u 'tracium-backup@tracium-postgres.service' -f

Check PostgreSQL backup integrity:

sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg list /backup-raid/borg/tracium-postgres
 
sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg check --archives-only --last 1 /backup-raid/borg/tracium-postgres

Add a new source

Create a config:

sudo cp /etc/tracium-backup/sources.d/1c-filebase.conf.example \
  /etc/tracium-backup/sources.d/1c-filebase.conf
sudoedit /etc/tracium-backup/sources.d/1c-filebase.conf
sudo systemctl enable --now 'tracium-backup@1c-filebase.timer'

Run once manually:

sudo systemctl start 'tracium-backup@1c-filebase.service'
journalctl -u 'tracium-backup@1c-filebase.service' -n 200 --no-pager

For 1C:

file-base mode: stop/quiesce 1C in PRE_HOOK, copy files, start in POST_HOOK;
SQL-backed mode: use a dump/export hook and back up the dump staging directory, not live database files.

Inspect backups

source=host-ssd
 
sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg list "/backup-raid/borg/$source"
 
latest=$(sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg list --short "/backup-raid/borg/$source" | tail -1)
 
sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg info "/backup-raid/borg/$source::$latest"

Check the latest archive:

source=host-ssd
 
sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg check --archives-only --last 1 "/backup-raid/borg/$source"

Restore probe:

latest=$(sudo BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg list --short /backup-raid/borg/host-ssd | tail -1)
sudo rm -rf /backup-raid/restore-test/host-ssd-latest
sudo mkdir -p /backup-raid/restore-test/host-ssd-latest
sudo env latest="$latest" BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  bash -c 'cd /backup-raid/restore-test/host-ssd-latest && borg extract "/backup-raid/borg/host-ssd::$latest" etc/hostname'
sudo cmp -s /etc/hostname /backup-raid/restore-test/host-ssd-latest/etc/hostname

RAID operations

Status:

cat /proc/mdstat
sudo mdadm --detail /dev/md/backup12t
findmnt /backup-raid
df -hT /backup-raid

Disk health:

sudo smartctl -H /dev/sdb
sudo smartctl -H /dev/sdc
sudo smartctl -H /dev/sda
sudo smartctl -H /dev/nvme0n1

If a disk fails, replace only the failed member. Identify by serial before any destructive command:

lsblk -e7 -o NAME,SIZE,MODEL,SERIAL,FSTYPE,MOUNTPOINTS
sudo mdadm --detail /dev/md/backup12t

After replacing a failed disk, partition it like the healthy member, then add it:

sudo sfdisk -d /dev/sdX | sudo sfdisk /dev/sdY
sudo partprobe /dev/sdY
sudo mdadm --add /dev/md/backup12t /dev/sdY1
watch cat /proc/mdstat

Use the actual device names from mdadm --detail; do not assume sdX/sdY.

Important caveats

Borg repositories are local and unencrypted. This is intentional for unattended local restore, but physical access to the disks is equivalent to access to the backup contents.
RAID1 is not a backup by itself; it protects against one HDD failure. Borg archives provide historical restore points.
The current /backup 1 TB disk is still live storage. The example legacy-backup-1tb.conf.example can snapshot it into Borg, but enabling it should be a separate operator decision because the first run may copy hundreds of GB.
Active DB files must not be backed up by plain file copy. Add dedicated dump sources for Postgres, 1C SQL, or other databases.

Tracium

Разделы