# Homelab Operational Scripts

This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure.

## Quick Navigation

- **[Monitoring & Maintenance](#monitoring--maintenance)** - Docker prune watchdog, backup hooks
- **[Network Configuration](#network-configuration)** - MTU migration for Jumbo Frames
- **[Deployment & Fixes](#deployment--fixes)** - Boot time fixes, systemd configuration

---

## Monitoring & Maintenance

### 1. Docker Prune Watchdog

**Purpose**: Automatically manages disk space on Docker nodes to prevent "disk full" failures

**Files**:
- `scripts/monitoring/docker-prune-watchdog.sh` - Main watchdog script
- `scripts/monitoring/docker-prune-watchdog.env.example` - Configuration template
- `scripts/monitoring/docker-prune-watchdog.service` - Systemd unit

**Deployment**:
```bash
# On each docker LXC (docker-1, docker-2, docker-3):
sudo cp docker-prune-watchdog.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh
sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env
sudo cp docker-prune-watchdog.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable docker-prune-watchdog.service
sudo systemctl start docker-prune-watchdog.service
```

**Configuration**:
Edit `/etc/docker-prune-watchdog.env`:
```ini
# Disk usage threshold (%) that triggers prune
DISK_THRESHOLD=85

# Check interval in seconds
CHECK_INTERVAL=60

# Notification mode: always, errors_only, both
NOTIFICATION_MODE=both

# Microsoft Teams webhook URL (for notifications)
TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/...
```

**Verification**:
```bash
# Check status
systemctl status docker-prune-watchdog.service

# View logs
cat /dev/shm/docker-prune-watchdog.log
tail -f /dev/shm/docker-prune-watchdog.log

# Manual check
df -h /volume1/docker-root
```

**Related Documentation**: ADR-011 (Observability & Self-Healing)

---

### 2. Proxmox Backup Hook

**Purpose**: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption

**File**: `scripts/backup/vzdump-docker-hook.sh`

**Installation**:
```bash
# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3):
sudo cp vzdump-docker-hook.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh
```

**Configuration**:
In `/etc/vzdump.conf` or per backup job, add:
```
script: /usr/local/bin/vzdump-docker-hook.sh
```

**What it does**:
1. **On backup-start**: 
   - Drains Docker Swarm node
   - Gracefully stops Docker service
   - Sends Teams notification

2. **On post-restart**:
   - Restarts Docker service
   - Re-activates Swarm node
   - Sends Teams notification

3. **On backup-abort**:
   - Immediately restarts Docker
   - Restores services
   - Sends warning notification

**Docker LXC IDs**: `4031` (docker-1), `4032` (docker-2), `4033` (docker-3)

**Verification**:
```bash
# Check hook logs on Proxmox node
tail -f /var/log/vzdump-docker-hook.log

# Monitor backup job
# (Check Proxmox backup status in UI)
```

**Known Bugs (Fixed in April 2026)**:
- Unbound variable `$2` when Proxmox calls with `job-init` phase
- Fix: Changed `VMID="$2"` to `VMID="${2:-}"`

**Related Documentation**: ADR-009 (Backups)

---

## Network Configuration

### 3. MTU 9000 Jumbo Frames Migration

**Purpose**: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput

**Files**:
- `scripts/network/mtu_phase1.sh` - Write daemon.json, restart Docker
- `scripts/network/mtu_phase2.sh` - Recreate overlay networks, redeploy stacks
- `scripts/network/mtu_rollback.sh` - Restore original configuration

**Prerequisites**:
- Proxmox host MTU already set to 9000
- LXC net0 config already has `mtu=9000`
- All scripts placed in `/volume1/docker/mtu_migration/` on each docker LXC

**Deployment Steps**:

1. **Copy scripts to CephFS** (accessible by all 3 docker LXCs):
```bash
# From any docker LXC:
mkdir -p /volume1/docker/mtu_migration
cp mtu_phase1.sh /volume1/docker/mtu_migration/
cp mtu_phase2.sh /volume1/docker/mtu_migration/
cp mtu_rollback.sh /volume1/docker/mtu_migration/
chmod +x /volume1/docker/mtu_migration/mtu_*.sh
```

2. **Phase 1** (synchronized on all nodes):
```bash
# Calculate sync epoch (T+3min from now, e.g., 1755423180)
SYNC_EPOCH=$(( $(date +%s) + 180 ))

# On docker-1:
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH &

# On docker-2:
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH &

# On docker-3:
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH &
```

3. **Phase 2** (stack redeployment):
```bash
# On docker-2 (or any node with swarm leadership):
nohup /volume1/docker/mtu_migration/mtu_phase2.sh &
tail -f /volume1/docker/mtu_migration/phase2.log
```

**Monitoring Progress**:
```bash
# Check phase1 completion:
ls /volume1/docker/mtu_migration/*_phase1.done

# View logs:
cat /volume1/docker/mtu_migration/docker-1_phase1.log
cat /volume1/docker/mtu_migration/docker-2_phase1.log
cat /volume1/docker/mtu_migration/docker-3_phase1.log

# View phase2 progress:
tail -f /volume1/docker/mtu_migration/phase2.log

# Verify MTU on running interface:
ip link show eth0 | grep mtu

# Verify overlay network MTUs:
docker network ls --filter driver=overlay -q | \
  xargs docker network inspect --format '{{.Name}} {{json .Options}}'
```

**Rollback**:
```bash
# On each docker LXC:
/volume1/docker/mtu_migration/mtu_rollback.sh

# Then redeploy stacks:
cd /volume1/docker/compose-files
docker stack deploy -c traefik.yaml traefik
docker stack deploy -c auth.yaml auth
docker stack deploy -c postgresql.yaml postgresql
```

**Related Documentation**: ADR-006 (Jumbo Frames)

---

## Deployment & Fixes

### 4. rc.local Boot Shebang Fix

**File**: `scripts/fixes/rc.local.fix.sh`

**Issue**: `/etc/rc.local` missing `#!/bin/bash` shebang causes boot errors

**Fix** (on docker-3):
```bash
# Add shebang to first line
sed -i '1i#!/bin/bash' /etc/rc.local
chmod +x /etc/rc.local

# Verify:
head -1 /etc/rc.local
```

---

### 5. systemd-networkd-wait-online Service Fix

**Issue**: Service waits indefinitely for all network interfaces, causes LXC boot delays

**Fix** (on docker-3):
```bash
# Mask the service
systemctl mask systemd-networkd-wait-online.service
systemctl daemon-reload

# Verify:
systemctl is-enabled systemd-networkd-wait-online.service
# Should show: masked
```

---

## Directory Structure

```
scripts/
├── monitoring/
│   ├── docker-prune-watchdog.sh
│   ├── docker-prune-watchdog.env.example
│   └── docker-prune-watchdog.service
├── backup/
│   └── vzdump-docker-hook.sh
├── network/
│   ├── mtu_phase1.sh
│   ├── mtu_phase2.sh
│   └── mtu_rollback.sh
├── fixes/
│   └── rc.local.fix.sh
└── README.md (this file)
```

---

## Related Architecture Decision Records (ADRs)

- **ADR-006**: Jumbo Frames (MTU 9000) for Docker Swarm
- **ADR-009**: Backup strategy and hook scripts
- **ADR-011**: Observability & Self-Healing (prune watchdog, Uptime Kuma)

---

## Key Notes

### Teams Webhook Deprecation
The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to:
- Power Automate flow (HTTP trigger → Post to Teams channel)
- Update all webhook URLs accordingly

### Disk Prune Aggressiveness
`docker system prune -f` is aggressive:
- Removes all stopped containers ✓
- Removes unused networks ✓
- Removes dangling images ✓
- Removes build cache ✓
- Does NOT remove volumes (safe) ✓
- Removes non-running services' images (can cause re-pulls)

**Mitigation**: Pin critical image versions in compose files (avoid `latest` tags)

### CephFS Shared Scripts
All MTU migration scripts are on shared CephFS at `/volume1/docker/mtu_migration/`, visible to all 3 docker LXCs.

---

## Troubleshooting

### Prune watchdog not triggering
```bash
# Check configuration
cat /etc/docker-prune-watchdog.env

# Verify service is running
systemctl status docker-prune-watchdog.service

# Check actual disk usage
df -h /volume1/docker-root

# Manually trigger prune if over threshold
docker system prune -f
```

### Backup hook errors
```bash
# Verify hook is installed
ls -la /usr/local/bin/vzdump-docker-hook.sh

# Check logs
tail -f /var/log/vzdump-docker-hook.log

# Test manually
/usr/local/bin/vzdump-docker-hook.sh job-init

# Verify Docker can be stopped/started
systemctl stop docker
systemctl start docker
```

### MTU migration stuck
```bash
# Check what phase completed
ls /volume1/docker/mtu_migration/*.done

# View logs
cat /volume1/docker/mtu_migration/docker-X_phase1.log
cat /volume1/docker/mtu_migration/phase2.log

# Check swarm status
docker node ls

# Verify interfaces
ip link show eth0
ip link show vmbr0

# If needed, manually complete remaining phases
# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook
```

---

## Contact & Support

For questions about these scripts, refer to:
- Implementation notes in chat history
- Architecture Decision Records (ADRs)
- Proxmox & MCP Setup documentation