diff --git a/SCRIPTS-README.md b/SCRIPTS-README.md deleted file mode 100644 index 289ff25..0000000 --- a/SCRIPTS-README.md +++ /dev/null @@ -1,359 +0,0 @@ -# Homelab Operational Scripts - -This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure. - -## Quick Navigation - -- **[Monitoring & Maintenance](#monitoring--maintenance)** - Docker prune watchdog, backup hooks -- **[Network Configuration](#network-configuration)** - MTU migration for Jumbo Frames -- **[Deployment & Fixes](#deployment--fixes)** - Boot time fixes, systemd configuration - ---- - -## Monitoring & Maintenance - -### 1. Docker Prune Watchdog - -**Purpose**: Automatically manages disk space on Docker nodes to prevent "disk full" failures - -**Files**: -- `scripts/monitoring/docker-prune-watchdog.sh` - Main watchdog script -- `scripts/monitoring/docker-prune-watchdog.env.example` - Configuration template -- `scripts/monitoring/docker-prune-watchdog.service` - Systemd unit - -**Deployment**: -```bash -# On each docker LXC (docker-1, docker-2, docker-3): -sudo cp docker-prune-watchdog.sh /usr/local/bin/ -sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh -sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env -sudo cp docker-prune-watchdog.service /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable docker-prune-watchdog.service -sudo systemctl start docker-prune-watchdog.service -``` - -**Configuration**: -Edit `/etc/docker-prune-watchdog.env`: -```ini -# Disk usage threshold (%) that triggers prune -DISK_THRESHOLD=85 - -# Check interval in seconds -CHECK_INTERVAL=60 - -# Notification mode: always, errors_only, both -NOTIFICATION_MODE=both - -# Microsoft Teams webhook URL (for notifications) -TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/... -``` - -**Verification**: -```bash -# Check status -systemctl status docker-prune-watchdog.service - -# View logs -cat /dev/shm/docker-prune-watchdog.log -tail -f /dev/shm/docker-prune-watchdog.log - -# Manual check -df -h /volume1/docker-root -``` - -**Related Documentation**: ADR-011 (Observability & Self-Healing) - ---- - -### 2. Proxmox Backup Hook - -**Purpose**: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption - -**File**: `scripts/backup/vzdump-docker-hook.sh` - -**Installation**: -```bash -# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3): -sudo cp vzdump-docker-hook.sh /usr/local/bin/ -sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh -``` - -**Configuration**: -In `/etc/vzdump.conf` or per backup job, add: -``` -script: /usr/local/bin/vzdump-docker-hook.sh -``` - -**What it does**: -1. **On backup-start**: - - Drains Docker Swarm node - - Gracefully stops Docker service - - Sends Teams notification - -2. **On post-restart**: - - Restarts Docker service - - Re-activates Swarm node - - Sends Teams notification - -3. **On backup-abort**: - - Immediately restarts Docker - - Restores services - - Sends warning notification - -**Docker LXC IDs**: `4031` (docker-1), `4032` (docker-2), `4033` (docker-3) - -**Verification**: -```bash -# Check hook logs on Proxmox node -tail -f /var/log/vzdump-docker-hook.log - -# Monitor backup job -# (Check Proxmox backup status in UI) -``` - -**Known Bugs (Fixed in April 2026)**: -- Unbound variable `$2` when Proxmox calls with `job-init` phase -- Fix: Changed `VMID="$2"` to `VMID="${2:-}"` - -**Related Documentation**: ADR-009 (Backups) - ---- - -## Network Configuration - -### 3. MTU 9000 Jumbo Frames Migration - -**Purpose**: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput - -**Files**: -- `scripts/network/mtu_phase1.sh` - Write daemon.json, restart Docker -- `scripts/network/mtu_phase2.sh` - Recreate overlay networks, redeploy stacks -- `scripts/network/mtu_rollback.sh` - Restore original configuration - -**Prerequisites**: -- Proxmox host MTU already set to 9000 -- LXC net0 config already has `mtu=9000` -- All scripts placed in `/volume1/docker/mtu_migration/` on each docker LXC - -**Deployment Steps**: - -1. **Copy scripts to CephFS** (accessible by all 3 docker LXCs): -```bash -# From any docker LXC: -mkdir -p /volume1/docker/mtu_migration -cp mtu_phase1.sh /volume1/docker/mtu_migration/ -cp mtu_phase2.sh /volume1/docker/mtu_migration/ -cp mtu_rollback.sh /volume1/docker/mtu_migration/ -chmod +x /volume1/docker/mtu_migration/mtu_*.sh -``` - -2. **Phase 1** (synchronized on all nodes): -```bash -# Calculate sync epoch (T+3min from now, e.g., 1755423180) -SYNC_EPOCH=$(( $(date +%s) + 180 )) - -# On docker-1: -nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH & - -# On docker-2: -nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH & - -# On docker-3: -nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH & -``` - -3. **Phase 2** (stack redeployment): -```bash -# On docker-2 (or any node with swarm leadership): -nohup /volume1/docker/mtu_migration/mtu_phase2.sh & -tail -f /volume1/docker/mtu_migration/phase2.log -``` - -**Monitoring Progress**: -```bash -# Check phase1 completion: -ls /volume1/docker/mtu_migration/*_phase1.done - -# View logs: -cat /volume1/docker/mtu_migration/docker-1_phase1.log -cat /volume1/docker/mtu_migration/docker-2_phase1.log -cat /volume1/docker/mtu_migration/docker-3_phase1.log - -# View phase2 progress: -tail -f /volume1/docker/mtu_migration/phase2.log - -# Verify MTU on running interface: -ip link show eth0 | grep mtu - -# Verify overlay network MTUs: -docker network ls --filter driver=overlay -q | \ - xargs docker network inspect --format '{{.Name}} {{json .Options}}' -``` - -**Rollback**: -```bash -# On each docker LXC: -/volume1/docker/mtu_migration/mtu_rollback.sh - -# Then redeploy stacks: -cd /volume1/docker/compose-files -docker stack deploy -c traefik.yaml traefik -docker stack deploy -c auth.yaml auth -docker stack deploy -c postgresql.yaml postgresql -``` - -**Related Documentation**: ADR-006 (Jumbo Frames) - ---- - -## Deployment & Fixes - -### 4. rc.local Boot Shebang Fix - -**File**: `scripts/fixes/rc.local.fix.sh` - -**Issue**: `/etc/rc.local` missing `#!/bin/bash` shebang causes boot errors - -**Fix** (on docker-3): -```bash -# Add shebang to first line -sed -i '1i#!/bin/bash' /etc/rc.local -chmod +x /etc/rc.local - -# Verify: -head -1 /etc/rc.local -``` - ---- - -### 5. systemd-networkd-wait-online Service Fix - -**Issue**: Service waits indefinitely for all network interfaces, causes LXC boot delays - -**Fix** (on docker-3): -```bash -# Mask the service -systemctl mask systemd-networkd-wait-online.service -systemctl daemon-reload - -# Verify: -systemctl is-enabled systemd-networkd-wait-online.service -# Should show: masked -``` - ---- - -## Directory Structure - -``` -scripts/ -├── monitoring/ -│ ├── docker-prune-watchdog.sh -│ ├── docker-prune-watchdog.env.example -│ └── docker-prune-watchdog.service -├── backup/ -│ └── vzdump-docker-hook.sh -├── network/ -│ ├── mtu_phase1.sh -│ ├── mtu_phase2.sh -│ └── mtu_rollback.sh -├── fixes/ -│ └── rc.local.fix.sh -└── README.md (this file) -``` - ---- - -## Related Architecture Decision Records (ADRs) - -- **ADR-006**: Jumbo Frames (MTU 9000) for Docker Swarm -- **ADR-009**: Backup strategy and hook scripts -- **ADR-011**: Observability & Self-Healing (prune watchdog, Uptime Kuma) - ---- - -## Key Notes - -### Teams Webhook Deprecation -The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to: -- Power Automate flow (HTTP trigger → Post to Teams channel) -- Update all webhook URLs accordingly - -### Disk Prune Aggressiveness -`docker system prune -f` is aggressive: -- Removes all stopped containers ✓ -- Removes unused networks ✓ -- Removes dangling images ✓ -- Removes build cache ✓ -- Does NOT remove volumes (safe) ✓ -- Removes non-running services' images (can cause re-pulls) - -**Mitigation**: Pin critical image versions in compose files (avoid `latest` tags) - -### CephFS Shared Scripts -All MTU migration scripts are on shared CephFS at `/volume1/docker/mtu_migration/`, visible to all 3 docker LXCs. - ---- - -## Troubleshooting - -### Prune watchdog not triggering -```bash -# Check configuration -cat /etc/docker-prune-watchdog.env - -# Verify service is running -systemctl status docker-prune-watchdog.service - -# Check actual disk usage -df -h /volume1/docker-root - -# Manually trigger prune if over threshold -docker system prune -f -``` - -### Backup hook errors -```bash -# Verify hook is installed -ls -la /usr/local/bin/vzdump-docker-hook.sh - -# Check logs -tail -f /var/log/vzdump-docker-hook.log - -# Test manually -/usr/local/bin/vzdump-docker-hook.sh job-init - -# Verify Docker can be stopped/started -systemctl stop docker -systemctl start docker -``` - -### MTU migration stuck -```bash -# Check what phase completed -ls /volume1/docker/mtu_migration/*.done - -# View logs -cat /volume1/docker/mtu_migration/docker-X_phase1.log -cat /volume1/docker/mtu_migration/phase2.log - -# Check swarm status -docker node ls - -# Verify interfaces -ip link show eth0 -ip link show vmbr0 - -# If needed, manually complete remaining phases -# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook -``` - ---- - -## Contact & Support - -For questions about these scripts, refer to: -- Implementation notes in chat history -- Architecture Decision Records (ADRs) -- Proxmox & MCP Setup documentation