diff --git a/SCRIPTS-README.md b/SCRIPTS-README.md new file mode 100644 index 0000000..289ff25 --- /dev/null +++ b/SCRIPTS-README.md @@ -0,0 +1,359 @@ +# Homelab Operational Scripts + +This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure. + +## Quick Navigation + +- **[Monitoring & Maintenance](#monitoring--maintenance)** - Docker prune watchdog, backup hooks +- **[Network Configuration](#network-configuration)** - MTU migration for Jumbo Frames +- **[Deployment & Fixes](#deployment--fixes)** - Boot time fixes, systemd configuration + +--- + +## Monitoring & Maintenance + +### 1. Docker Prune Watchdog + +**Purpose**: Automatically manages disk space on Docker nodes to prevent "disk full" failures + +**Files**: +- `scripts/monitoring/docker-prune-watchdog.sh` - Main watchdog script +- `scripts/monitoring/docker-prune-watchdog.env.example` - Configuration template +- `scripts/monitoring/docker-prune-watchdog.service` - Systemd unit + +**Deployment**: +```bash +# On each docker LXC (docker-1, docker-2, docker-3): +sudo cp docker-prune-watchdog.sh /usr/local/bin/ +sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh +sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env +sudo cp docker-prune-watchdog.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable docker-prune-watchdog.service +sudo systemctl start docker-prune-watchdog.service +``` + +**Configuration**: +Edit `/etc/docker-prune-watchdog.env`: +```ini +# Disk usage threshold (%) that triggers prune +DISK_THRESHOLD=85 + +# Check interval in seconds +CHECK_INTERVAL=60 + +# Notification mode: always, errors_only, both +NOTIFICATION_MODE=both + +# Microsoft Teams webhook URL (for notifications) +TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/... +``` + +**Verification**: +```bash +# Check status +systemctl status docker-prune-watchdog.service + +# View logs +cat /dev/shm/docker-prune-watchdog.log +tail -f /dev/shm/docker-prune-watchdog.log + +# Manual check +df -h /volume1/docker-root +``` + +**Related Documentation**: ADR-011 (Observability & Self-Healing) + +--- + +### 2. Proxmox Backup Hook + +**Purpose**: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption + +**File**: `scripts/backup/vzdump-docker-hook.sh` + +**Installation**: +```bash +# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3): +sudo cp vzdump-docker-hook.sh /usr/local/bin/ +sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh +``` + +**Configuration**: +In `/etc/vzdump.conf` or per backup job, add: +``` +script: /usr/local/bin/vzdump-docker-hook.sh +``` + +**What it does**: +1. **On backup-start**: + - Drains Docker Swarm node + - Gracefully stops Docker service + - Sends Teams notification + +2. **On post-restart**: + - Restarts Docker service + - Re-activates Swarm node + - Sends Teams notification + +3. **On backup-abort**: + - Immediately restarts Docker + - Restores services + - Sends warning notification + +**Docker LXC IDs**: `4031` (docker-1), `4032` (docker-2), `4033` (docker-3) + +**Verification**: +```bash +# Check hook logs on Proxmox node +tail -f /var/log/vzdump-docker-hook.log + +# Monitor backup job +# (Check Proxmox backup status in UI) +``` + +**Known Bugs (Fixed in April 2026)**: +- Unbound variable `$2` when Proxmox calls with `job-init` phase +- Fix: Changed `VMID="$2"` to `VMID="${2:-}"` + +**Related Documentation**: ADR-009 (Backups) + +--- + +## Network Configuration + +### 3. MTU 9000 Jumbo Frames Migration + +**Purpose**: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput + +**Files**: +- `scripts/network/mtu_phase1.sh` - Write daemon.json, restart Docker +- `scripts/network/mtu_phase2.sh` - Recreate overlay networks, redeploy stacks +- `scripts/network/mtu_rollback.sh` - Restore original configuration + +**Prerequisites**: +- Proxmox host MTU already set to 9000 +- LXC net0 config already has `mtu=9000` +- All scripts placed in `/volume1/docker/mtu_migration/` on each docker LXC + +**Deployment Steps**: + +1. **Copy scripts to CephFS** (accessible by all 3 docker LXCs): +```bash +# From any docker LXC: +mkdir -p /volume1/docker/mtu_migration +cp mtu_phase1.sh /volume1/docker/mtu_migration/ +cp mtu_phase2.sh /volume1/docker/mtu_migration/ +cp mtu_rollback.sh /volume1/docker/mtu_migration/ +chmod +x /volume1/docker/mtu_migration/mtu_*.sh +``` + +2. **Phase 1** (synchronized on all nodes): +```bash +# Calculate sync epoch (T+3min from now, e.g., 1755423180) +SYNC_EPOCH=$(( $(date +%s) + 180 )) + +# On docker-1: +nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH & + +# On docker-2: +nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH & + +# On docker-3: +nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH & +``` + +3. **Phase 2** (stack redeployment): +```bash +# On docker-2 (or any node with swarm leadership): +nohup /volume1/docker/mtu_migration/mtu_phase2.sh & +tail -f /volume1/docker/mtu_migration/phase2.log +``` + +**Monitoring Progress**: +```bash +# Check phase1 completion: +ls /volume1/docker/mtu_migration/*_phase1.done + +# View logs: +cat /volume1/docker/mtu_migration/docker-1_phase1.log +cat /volume1/docker/mtu_migration/docker-2_phase1.log +cat /volume1/docker/mtu_migration/docker-3_phase1.log + +# View phase2 progress: +tail -f /volume1/docker/mtu_migration/phase2.log + +# Verify MTU on running interface: +ip link show eth0 | grep mtu + +# Verify overlay network MTUs: +docker network ls --filter driver=overlay -q | \ + xargs docker network inspect --format '{{.Name}} {{json .Options}}' +``` + +**Rollback**: +```bash +# On each docker LXC: +/volume1/docker/mtu_migration/mtu_rollback.sh + +# Then redeploy stacks: +cd /volume1/docker/compose-files +docker stack deploy -c traefik.yaml traefik +docker stack deploy -c auth.yaml auth +docker stack deploy -c postgresql.yaml postgresql +``` + +**Related Documentation**: ADR-006 (Jumbo Frames) + +--- + +## Deployment & Fixes + +### 4. rc.local Boot Shebang Fix + +**File**: `scripts/fixes/rc.local.fix.sh` + +**Issue**: `/etc/rc.local` missing `#!/bin/bash` shebang causes boot errors + +**Fix** (on docker-3): +```bash +# Add shebang to first line +sed -i '1i#!/bin/bash' /etc/rc.local +chmod +x /etc/rc.local + +# Verify: +head -1 /etc/rc.local +``` + +--- + +### 5. systemd-networkd-wait-online Service Fix + +**Issue**: Service waits indefinitely for all network interfaces, causes LXC boot delays + +**Fix** (on docker-3): +```bash +# Mask the service +systemctl mask systemd-networkd-wait-online.service +systemctl daemon-reload + +# Verify: +systemctl is-enabled systemd-networkd-wait-online.service +# Should show: masked +``` + +--- + +## Directory Structure + +``` +scripts/ +├── monitoring/ +│ ├── docker-prune-watchdog.sh +│ ├── docker-prune-watchdog.env.example +│ └── docker-prune-watchdog.service +├── backup/ +│ └── vzdump-docker-hook.sh +├── network/ +│ ├── mtu_phase1.sh +│ ├── mtu_phase2.sh +│ └── mtu_rollback.sh +├── fixes/ +│ └── rc.local.fix.sh +└── README.md (this file) +``` + +--- + +## Related Architecture Decision Records (ADRs) + +- **ADR-006**: Jumbo Frames (MTU 9000) for Docker Swarm +- **ADR-009**: Backup strategy and hook scripts +- **ADR-011**: Observability & Self-Healing (prune watchdog, Uptime Kuma) + +--- + +## Key Notes + +### Teams Webhook Deprecation +The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to: +- Power Automate flow (HTTP trigger → Post to Teams channel) +- Update all webhook URLs accordingly + +### Disk Prune Aggressiveness +`docker system prune -f` is aggressive: +- Removes all stopped containers ✓ +- Removes unused networks ✓ +- Removes dangling images ✓ +- Removes build cache ✓ +- Does NOT remove volumes (safe) ✓ +- Removes non-running services' images (can cause re-pulls) + +**Mitigation**: Pin critical image versions in compose files (avoid `latest` tags) + +### CephFS Shared Scripts +All MTU migration scripts are on shared CephFS at `/volume1/docker/mtu_migration/`, visible to all 3 docker LXCs. + +--- + +## Troubleshooting + +### Prune watchdog not triggering +```bash +# Check configuration +cat /etc/docker-prune-watchdog.env + +# Verify service is running +systemctl status docker-prune-watchdog.service + +# Check actual disk usage +df -h /volume1/docker-root + +# Manually trigger prune if over threshold +docker system prune -f +``` + +### Backup hook errors +```bash +# Verify hook is installed +ls -la /usr/local/bin/vzdump-docker-hook.sh + +# Check logs +tail -f /var/log/vzdump-docker-hook.log + +# Test manually +/usr/local/bin/vzdump-docker-hook.sh job-init + +# Verify Docker can be stopped/started +systemctl stop docker +systemctl start docker +``` + +### MTU migration stuck +```bash +# Check what phase completed +ls /volume1/docker/mtu_migration/*.done + +# View logs +cat /volume1/docker/mtu_migration/docker-X_phase1.log +cat /volume1/docker/mtu_migration/phase2.log + +# Check swarm status +docker node ls + +# Verify interfaces +ip link show eth0 +ip link show vmbr0 + +# If needed, manually complete remaining phases +# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook +``` + +--- + +## Contact & Support + +For questions about these scripts, refer to: +- Implementation notes in chat history +- Architecture Decision Records (ADRs) +- Proxmox & MCP Setup documentation