# Backup & Recovery Procedures

**Project:** LCBP3-DMS
**Version:** 1.6.0
**Last Updated:** 2025-12-02

---

## 📋 Overview

This document outlines backup strategies, recovery procedures, and disaster recovery planning for LCBP3-DMS.

---

## 🎯 Backup Strategy

### Backup Schedule

| Data Type              | Frequency      | Retention | Method                  |
| ---------------------- | -------------- | --------- | ----------------------- |
| Database (Full)        | Daily at 02:00 | 30 days   | mysqldump + compression |
| Database (Incremental) | Every 6 hours  | 7 days    | Binary logs             |
| File Uploads           | Daily at 03:00 | 30 days   | rsync to backup server  |
| Configuration Files    | Weekly         | 90 days   | Git repository          |
| Elasticsearch Indexes  | Weekly         | 14 days   | Snapshot to S3/NFS      |
| Application Logs       | Daily          | 90 days   | Rotation + archival     |

### Backup Locations

**Primary Backup:** QNAP NAS `/backup/lcbp3-dms`
**Secondary Backup:** External backup server (rsync)
**Offsite Backup:** Cloud storage (optional - for critical data)

---

## 💾 Database Backup

### Automated Daily Backup Script

```bash
#!/bin/bash
# File: /scripts/backup-database.sh

# Configuration
BACKUP_DIR="/backup/lcbp3-dms/database"
DB_CONTAINER="lcbp3-mariadb"
DB_NAME="lcbp3_dms"
DB_USER="backup_user"
DB_PASS="<BACKUP_USER_PASSWORD>"
RETENTION_DAYS=30

# Create backup directory
BACKUP_FILE="$BACKUP_DIR/lcbp3_$(date +%Y%m%d_%H%M%S).sql.gz"
mkdir -p "$BACKUP_DIR"

# Perform backup
echo "Starting database backup to $BACKUP_FILE"
docker exec $DB_CONTAINER mysqldump \
  --user=$DB_USER \
  --password=$DB_PASS \
  --single-transaction \
  --routines \
  --triggers \
  --databases $DB_NAME \
  | gzip > "$BACKUP_FILE"

# Check backup success
if [ $? -eq 0 ]; then
  echo "Backup completed successfully"

  # Delete old backups
  find "$BACKUP_DIR" -name "*.sql.gz" -type f -mtime +$RETENTION_DAYS -delete
  echo "Old backups cleaned up (retention: $RETENTION_DAYS days)"
else
  echo "ERROR: Backup failed!"
  exit 1
fi
```

### Schedule with Cron

```bash
# Edit crontab
crontab -e

# Add backup job (runs daily at 2 AM)
0 2 * * * /scripts/backup-database.sh >> /var/log/backup-database.log 2>&1
```

### Manual Database Backup

```bash
# Backup specific database
docker exec lcbp3-mariadb mysqldump \
  -u root -p \
  --single-transaction \
  lcbp3_dms > backup_$(date +%Y%m%d).sql

# Compress backup
gzip backup_$(date +%Y%m%d).sql
```

---

## 📂 File Uploads Backup

### Automated Rsync Backup

```bash
#!/bin/bash
# File: /scripts/backup-uploads.sh

SOURCE="/var/lib/docker/volumes/lcbp3_uploads/_data"
DEST="/backup/lcbp3-dms/uploads"
RETENTION_DAYS=30

# Create incremental backup with rsync
rsync -av --delete \
  --backup --backup-dir="$DEST/backup-$(date +%Y%m%d)" \
  "$SOURCE/" "$DEST/current/"

# Cleanup old backups
find "$DEST" -maxdepth 1 -type d -name "backup-*" -mtime +$RETENTION_DAYS -exec rm -rf {} \;

echo "Upload backup completed: $(date)"
```

### Schedule Uploads Backup

```bash
# Run daily at 3 AM
0 3 * * * /scripts/backup-uploads.sh >> /var/log/backup-uploads.log 2>&1
```

---

## 🔄 Database Recovery

### Full Database Restore

```bash
# Step 1: Stop backend application
docker stop lcbp3-backend

# Step 2: Restore database from backup
gunzip < backup_20241201.sql.gz | \
  docker exec -i lcbp3-mariadb mysql -u root -p lcbp3_dms

# Step 3: Verify restore
docker exec lcbp3-mariadb mysql -u root -p -e "
  USE lcbp3_dms;
  SELECT COUNT(*) FROM users;
  SELECT COUNT(*) FROM correspondences;
"

# Step 4: Restart backend
docker start lcbp3-backend
```

### Point-in-Time Recovery (Using Binary Logs)

```bash
# Step 1: Restore last full backup
gunzip < backup_20241201_020000.sql.gz | \
  docker exec -i lcbp3-mariadb mysql -u root -p lcbp3_dms

# Step 2: Apply binary logs since backup
docker exec lcbp3-mariadb mysqlbinlog \
  --start-datetime="2024-12-01 02:00:00" \
  --stop-datetime="2024-12-01 14:30:00" \
  /var/lib/mysql/mysql-bin.000001 | \
  docker exec -i lcbp3-mariadb mysql -u root -p lcbp3_dms
```

---

## 📁 File Uploads Recovery

### Restore from Backup

```bash
# Stop backend to prevent file operations
docker stop lcbp3-backend

# Restore files
rsync -av \
  /backup/lcbp3-dms/uploads/current/ \
  /var/lib/docker/volumes/lcbp3_uploads/_data/

# Verify permissions
docker exec lcbp3-backend chown -R node:node /app/uploads

# Restart backend
docker start lcbp3-backend
```

---

## 🚨 Disaster Recovery Plan

### RTO & RPO

- **RTO (Recovery Time Objective):** 4 hours
- **RPO (Recovery Point Objective):** 24 hours (for files), 6 hours (for database)

### DR Scenarios

#### Scenario 1: Database Corruption

**Detection:** Database errors in logs, application errors
**Recovery Time:** 30 minutes
**Steps:**

1. Stop backend
2. Restore last full backup
3. Apply binary logs (if needed)
4. Verify data integrity
5. Restart services

#### Scenario 2: Complete Server Failure

**Detection:** Server unresponsive
**Recovery Time:** 4 hours
**Steps:**

1. Provision new QNAP server or VM
2. Install Docker & Container Station
3. Clone Git repository
4. Restore database backup
5. Restore file uploads
6. Deploy containers
7. Update DNS (if needed)
8. Verify functionality

#### Scenario 3: Ransomware Attack

**Detection:** Encrypted files, ransom note
**Recovery Time:** 6 hours
**Steps:**

1. **DO NOT pay ransom**
2. Isolate infected server
3. Provision clean environment
4. Restore from offsite backup
5. Scan restored backup for malware
6. Deploy and verify
7. Review security logs
8. Implement additional security measures

---

## ✅ Backup Verification

### Weekly Backup Testing

```bash
#!/bin/bash
# File: /scripts/test-backup.sh

# Create temporary test database
docker exec lcbp3-mariadb mysql -u root -p -e "
  CREATE DATABASE IF NOT EXISTS test_restore;
"

# Restore latest backup to test database
LATEST_BACKUP=$(ls -t /backup/lcbp3-dms/database/*.sql.gz | head -1)
gunzip < "$LATEST_BACKUP" | \
  sed 's/USE `lcbp3_dms`/USE `test_restore`/g' | \
  docker exec -i lcbp3-mariadb mysql -u root -p

# Verify table counts
docker exec lcbp3-mariadb mysql -u root -p -e "
  SELECT COUNT(*)  FROM test_restore.users;
  SELECT COUNT(*) FROM test_restore.correspondences;
"

# Cleanup
docker exec lcbp3-mariadb mysql -u root -p -e "
  DROP DATABASE test_restore;
"

echo "Backup verification completed: $(date)"
```

### Monthly DR Drill

- Test full system restore on standby server
- Document time taken and issues encountered
- Update DR procedures based on findings

---

## 📊 Backup Monitoring

### Backup Status Dashboard

Monitor:

- ✅ Last successful backup timestamp
- ✅ Backup file size (detect anomalies)
- ✅ Backup success/failure rate
- ✅ Available backup storage space

### Alerts

Send alert if:

- ❌ Backup fails
- ❌ Backup file size < 50% of average (possible corruption)
- ❌ No backup in last 48 hours
- ❌ Backup storage < 20% free

---

## 🔧 Maintenance

### Optimize Backup Performance

```sql
-- Enable InnoDB compression for large tables
ALTER TABLE correspondences ROW_FORMAT=COMPRESSED;
ALTER TABLE workflow_history ROW_FORMAT=COMPRESSED;

-- Archive old audit logs
-- Move records older than 1 year to archive table
INSERT INTO audit_logs_archive
SELECT * FROM audit_logs
WHERE created_at < DATE_SUB(NOW(), INTERVAL 1 YEAR);

DELETE FROM audit_logs
WHERE created_at < DATE_SUB(NOW(), INTERVAL 1 YEAR);
```

---

## 📚 Backup Checklist

### Daily Tasks

- [ ] Verify automated backups completed
- [ ] Check backup log files for errors
- [ ] Monitor backup storage space

### Weekly Tasks

- [ ] Test restore from random backup
- [ ] Review backup size trends
- [ ] Verify offsite backups synced

### Monthly Tasks

- [ ] Full DR drill
- [ ] Review and update DR procedures
- [ ] Test backup restoration on different server

### Quarterly Tasks

- [ ] Audit backup access controls
- [ ] Review backup retention policies
- [ ] Update backup documentation

---

## 🔗 Related Documents

- [Deployment Guide](04-01-deployment-guide.md)
- [Monitoring & Alerting](04-03-monitoring-alerting.md)
- [Incident Response](04-07-incident-response.md)

---

**Version:** 1.6.0
**Last Review:** 2025-12-01
**Next Review:** 2026-03-01
