Managing Database RecoveryCAUTION

From the repository's own README, as read at the audited commit. Badges and raw HTML are left out.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: managing-database-recovery
description: 'Process use when you need to work with database operations.

  This skill provides database management and optimization with comprehensive guidance
  and automation.

  Trigger with phrases like "manage database", "optimize database",

  or "configure database".

  '
allowed-tools: Read, Write, Edit, Grep, Glob, Bash(tar:*), Bash(rsync:*), Bash(aws:s3:*)
version: 1.26.0
author: Jeremy Longshore <[email protected]>
license: MIT
tags:
- database
- database-recovery
compatibility: Designed for Claude Code
---
# Database Recovery Manager

## Overview

Plan and execute database backup and recovery procedures for PostgreSQL and MySQL, including point-in-time recovery (PITR), logical and physical backups, WAL archiving, and disaster recovery testing. This skill covers the full backup lifecycle from configuration through automated verification, ensuring Recovery Point Objective (RPO) and Recovery Time Objective (RTO) targets are met.

## Prerequisites

- Database superuser or replication-role credentials
- Backup storage destination (local disk, NFS mount, S3, GCS, or Azure Blob)
- `pg_basebackup`, `pg_dump`, `pg_restore` (PostgreSQL) or `mysqldump`, `xtrabackup` (MySQL)
- `tar`, `rsync`, or `aws s3` CLI for backup transfer and storage
- WAL archiving configured for PITR (PostgreSQL: `archive_mode = on`, `archive_command`)
- Sufficient storage for backup retention (estimate 2-3x database size for full + incremental)

## Instructions

1. Assess the current backup situation by checking existing backup configurations. For PostgreSQL: verify `archive_mode`, `archive_command`, and `wal_level` in `postgresql.conf`. For MySQL: check if binary logging is enabled with `SHOW VARIABLES LIKE 'log_bin'`.

2. Define RPO and RTO targets based on business requirements:
   - RPO (acceptable data loss): determines backup frequency and WAL archiving interval
   - RTO (acceptable downtime): determines backup type and recovery procedure complexity
   - Typical targets: RPO < 1 hour (WAL archiving), RTO < 30 minutes (physical backup restore)

3. Configure WAL archiving for PostgreSQL PITR:
   - Set `wal_level = replica` and `archive_mode = on`
   - Configure `archive_command = 'test ! -f /archive/%f && cp %p /archive/%f'` (or use pgBackRest/WAL-G for S3)
   - Verify archiving works: `SELECT * FROM pg_stat_archiver`
   - For MySQL, enable binary logging: `log_bin = mysql-bin`, `binlog_format = ROW`

4. Create a full physical backup using `pg_basebackup -D /backups/base -Ft -z -P` (PostgreSQL) or `xtrabackup --backup --target-dir=/backups/full` (MySQL). Physical backups are faster to restore than logical backups for databases larger than 10GB.

5. Create logical backups for portability and selective restoration: `pg_dump -Fc -f database.dump dbname` (PostgreSQL) or `mysqldump --single-transaction --routines --triggers dbname > database.sql` (MySQL).

6. Upload backups to remote storage for disaster recovery: `aws s3 cp /backups/base.tar.gz s3://backup-bucket/postgres/$(date +%Y%m%d)/` with server-side encryption enabled. Implement a retention policy (e.g., daily backups for 30 days, weekly for 90 days, monthly for 1 year).

7. Test recovery by restoring to a separate server or container:
   - Restore physical backup: `pg_restore -d testdb database.dump` or untar base backup and start PostgreSQL
   - For PITR: restore base backup, copy WAL files to `pg_wal`, create `recovery.signal` with `recovery_target_time = '2024-01-15 14:30:00'`
   - Verify data integrity by running application test suite against restored database
   - Measure actual recovery time to validate RTO target

8. Automate backup verification with a daily cron job that: takes backup, restores to a test instance, runs integrity checks (`pg_catalog.pg_class` row counts, checksum verification), and sends a success/failure notification.

9. Document the recovery runbook with exact commands for each recovery scenario: full database restore, PITR to a specific timestamp, single table restore, and cross-region failover.

10. Schedule monthly disaster recovery drills to verify the runbook works and the team can execute recovery within RTO targets.

## Output

- **Backup configuration files** for PostgreSQL (postgresql.conf changes, archive_command) or MySQL (my.cnf changes)
- **Backup scripts** (shell) for automated full and incremental backups with S3/GCS upload
- **Recovery runbook** with step-by-step commands for each recovery scenario
- **Backup verification scripts** that automate restore-and-check procedures
- **Retention policy configuration** with automated cleanup of old backups

## Error Handling

| Error | Cause | Solution |
|-------|-------|---------|
| WAL segment not found during PITR | Gap in WAL archiving due to archive_command failure | Check `pg_stat_archiver` for `last_failed_wal`; fix archive_command; consider WAL-G or pgBackRest for reliable archiving |
| `pg_basebackup` fails with "replication connection" error | Missing replication permissions or `max_wal_senders` exhausted | Grant REPLICATION role; increase `max_wal_senders`; add entry to `pg_hba.conf` for replication connections |
| Backup storage full | Retention policy not enforced or backup size grew unexpectedly | Implement automated cleanup script; compress backups with `gzip` or `zstd`; monitor storage usage with alerts at 80% |
| Recovery takes longer than RTO target | Database grew since RTO was last validated, or restore is I/O-bound | Use physical backups instead of logical; restore to SSD storage; parallelize restore with `pg_restore -j 4`; consider standby replica for faster failover |
| Restored database has corruption | Backup taken during crash or disk error | Enable `data_checksums` in PostgreSQL; verify backups with `pg_verifybackup`; run `ANALYZE` and `REINDEX` after restore |

## Examples

**Point-in-time recovery after accidental table drop**: At 14:30 a developer runs `DROP TABLE orders` on production. Recover

CAUTIONgrade B · trust 89/100 Install with care. The audit found things worth knowing before you trust its output.

What the source does

Filesystem

none-observed

Network

declared (6 observation(s))

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (1)

VERIFY=false

Gates applied: no_behavioural_pass.