compose-files/SCRIPTS-README.md

Homelab Operational Scripts

This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure.

Quick Navigation

• Monitoring & Maintenance - Docker prune watchdog, backup hooks
• Network Configuration - MTU migration for Jumbo Frames
• Deployment & Fixes - Boot time fixes, systemd configuration

---

Monitoring & Maintenance

1. Docker Prune Watchdog

Purpose: Automatically manages disk space on Docker nodes to prevent "disk full" failures

Files:

• scripts/monitoring/docker-prune-watchdog.sh - Main watchdog script
• scripts/monitoring/docker-prune-watchdog.env.example - Configuration template
• scripts/monitoring/docker-prune-watchdog.service - Systemd unit

Deployment:

# On each docker LXC (docker-1, docker-2, docker-3):
sudo cp docker-prune-watchdog.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh
sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env
sudo cp docker-prune-watchdog.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable docker-prune-watchdog.service
sudo systemctl start docker-prune-watchdog.service

Configuration: Edit /etc/docker-prune-watchdog.env:

# Disk usage threshold (%) that triggers prune
DISK_THRESHOLD=85

# Check interval in seconds
CHECK_INTERVAL=60

# Notification mode: always, errors_only, both
NOTIFICATION_MODE=both

# Microsoft Teams webhook URL (for notifications)
TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/...

Verification:

# Check status
systemctl status docker-prune-watchdog.service

# View logs
cat /dev/shm/docker-prune-watchdog.log
tail -f /dev/shm/docker-prune-watchdog.log

# Manual check
df -h /volume1/docker-root

Related Documentation: ADR-011 (Observability & Self-Healing)

---

2. Proxmox Backup Hook

Purpose: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption

File: scripts/backup/vzdump-docker-hook.sh

Installation:

# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3):
sudo cp vzdump-docker-hook.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh

Configuration: In /etc/vzdump.conf or per backup job, add:

script: /usr/local/bin/vzdump-docker-hook.sh

What it does:

1. On backup-start:

   • Drains Docker Swarm node
   • Gracefully stops Docker service
   • Sends Teams notification

2. On post-restart:

   • Restarts Docker service
   • Re-activates Swarm node
   • Sends Teams notification

3. On backup-abort:

   • Immediately restarts Docker
   • Restores services
   • Sends warning notification

Docker LXC IDs: 4031 (docker-1), 4032 (docker-2), 4033 (docker-3)

Verification:

# Check hook logs on Proxmox node
tail -f /var/log/vzdump-docker-hook.log

# Monitor backup job
# (Check Proxmox backup status in UI)

Known Bugs (Fixed in April 2026):

• Unbound variable $2 when Proxmox calls with job-init phase
• Fix: Changed VMID="$2" to VMID="${2:-}"

Related Documentation: ADR-009 (Backups)

---

Network Configuration

3. MTU 9000 Jumbo Frames Migration

Purpose: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput

Files:

• scripts/network/mtu_phase1.sh - Write daemon.json, restart Docker
• scripts/network/mtu_phase2.sh - Recreate overlay networks, redeploy stacks
• scripts/network/mtu_rollback.sh - Restore original configuration

Prerequisites:

• Proxmox host MTU already set to 9000
• LXC net0 config already has mtu=9000
• All scripts placed in /volume1/docker/mtu_migration/ on each docker LXC

Deployment Steps:

1. Copy scripts to CephFS (accessible by all 3 docker LXCs):

# From any docker LXC:
mkdir -p /volume1/docker/mtu_migration
cp mtu_phase1.sh /volume1/docker/mtu_migration/
cp mtu_phase2.sh /volume1/docker/mtu_migration/
cp mtu_rollback.sh /volume1/docker/mtu_migration/
chmod +x /volume1/docker/mtu_migration/mtu_*.sh

2. Phase 1 (synchronized on all nodes):

# Calculate sync epoch (T+3min from now, e.g., 1755423180)
SYNC_EPOCH=$(( $(date +%s) + 180 ))

# On docker-1:
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH &

# On docker-2:
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH &

# On docker-3:
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH &

3. Phase 2 (stack redeployment):

# On docker-2 (or any node with swarm leadership):
nohup /volume1/docker/mtu_migration/mtu_phase2.sh &
tail -f /volume1/docker/mtu_migration/phase2.log

Monitoring Progress:

# Check phase1 completion:
ls /volume1/docker/mtu_migration/*_phase1.done

# View logs:
cat /volume1/docker/mtu_migration/docker-1_phase1.log
cat /volume1/docker/mtu_migration/docker-2_phase1.log
cat /volume1/docker/mtu_migration/docker-3_phase1.log

# View phase2 progress:
tail -f /volume1/docker/mtu_migration/phase2.log

# Verify MTU on running interface:
ip link show eth0 | grep mtu

# Verify overlay network MTUs:
docker network ls --filter driver=overlay -q | \
  xargs docker network inspect --format '{{.Name}} {{json .Options}}'

Rollback:

# On each docker LXC:
/volume1/docker/mtu_migration/mtu_rollback.sh

# Then redeploy stacks:
cd /volume1/docker/compose-files
docker stack deploy -c traefik.yaml traefik
docker stack deploy -c auth.yaml auth
docker stack deploy -c postgresql.yaml postgresql

Related Documentation: ADR-006 (Jumbo Frames)

---

Deployment & Fixes

4. rc.local Boot Shebang Fix

File: scripts/fixes/rc.local.fix.sh

Issue: /etc/rc.local missing #!/bin/bash shebang causes boot errors

Fix (on docker-3):

# Add shebang to first line
sed -i '1i#!/bin/bash' /etc/rc.local
chmod +x /etc/rc.local

# Verify:
head -1 /etc/rc.local

---

5. systemd-networkd-wait-online Service Fix

Issue: Service waits indefinitely for all network interfaces, causes LXC boot delays

Fix (on docker-3):

# Mask the service
systemctl mask systemd-networkd-wait-online.service
systemctl daemon-reload

# Verify:
systemctl is-enabled systemd-networkd-wait-online.service
# Should show: masked

---

Directory Structure

scripts/
├── monitoring/
│   ├── docker-prune-watchdog.sh
│   ├── docker-prune-watchdog.env.example
│   └── docker-prune-watchdog.service
├── backup/
│   └── vzdump-docker-hook.sh
├── network/
│   ├── mtu_phase1.sh
│   ├── mtu_phase2.sh
│   └── mtu_rollback.sh
├── fixes/
│   └── rc.local.fix.sh
└── README.md (this file)

---

Related Architecture Decision Records (ADRs)

• ADR-006: Jumbo Frames (MTU 9000) for Docker Swarm
• ADR-009: Backup strategy and hook scripts
• ADR-011: Observability & Self-Healing (prune watchdog, Uptime Kuma)

---

Key Notes

Teams Webhook Deprecation

The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to:

• Power Automate flow (HTTP trigger → Post to Teams channel)
• Update all webhook URLs accordingly

Disk Prune Aggressiveness

docker system prune -f is aggressive:

• Removes all stopped containers ✓
• Removes unused networks ✓
• Removes dangling images ✓
• Removes build cache ✓
• Does NOT remove volumes (safe) ✓
• Removes non-running services' images (can cause re-pulls)

Mitigation: Pin critical image versions in compose files (avoid latest tags)

CephFS Shared Scripts

All MTU migration scripts are on shared CephFS at /volume1/docker/mtu_migration/, visible to all 3 docker LXCs.

---

Troubleshooting

Prune watchdog not triggering

# Check configuration
cat /etc/docker-prune-watchdog.env

# Verify service is running
systemctl status docker-prune-watchdog.service

# Check actual disk usage
df -h /volume1/docker-root

# Manually trigger prune if over threshold
docker system prune -f

Backup hook errors

# Verify hook is installed
ls -la /usr/local/bin/vzdump-docker-hook.sh

# Check logs
tail -f /var/log/vzdump-docker-hook.log

# Test manually
/usr/local/bin/vzdump-docker-hook.sh job-init

# Verify Docker can be stopped/started
systemctl stop docker
systemctl start docker

MTU migration stuck

# Check what phase completed
ls /volume1/docker/mtu_migration/*.done

# View logs
cat /volume1/docker/mtu_migration/docker-X_phase1.log
cat /volume1/docker/mtu_migration/phase2.log

# Check swarm status
docker node ls

# Verify interfaces
ip link show eth0
ip link show vmbr0

# If needed, manually complete remaining phases
# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook

---

Contact & Support

For questions about these scripts, refer to:

• Implementation notes in chat history
• Architecture Decision Records (ADRs)
• Proxmox & MCP Setup documentation