Archived
360 lines
9.0 KiB
Markdown
360 lines
9.0 KiB
Markdown
# Homelab Operational Scripts
|
|
|
|
This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure.
|
|
|
|
## Quick Navigation
|
|
|
|
- **[Monitoring & Maintenance](#monitoring--maintenance)** - Docker prune watchdog, backup hooks
|
|
- **[Network Configuration](#network-configuration)** - MTU migration for Jumbo Frames
|
|
- **[Deployment & Fixes](#deployment--fixes)** - Boot time fixes, systemd configuration
|
|
|
|
---
|
|
|
|
## Monitoring & Maintenance
|
|
|
|
### 1. Docker Prune Watchdog
|
|
|
|
**Purpose**: Automatically manages disk space on Docker nodes to prevent "disk full" failures
|
|
|
|
**Files**:
|
|
- `scripts/monitoring/docker-prune-watchdog.sh` - Main watchdog script
|
|
- `scripts/monitoring/docker-prune-watchdog.env.example` - Configuration template
|
|
- `scripts/monitoring/docker-prune-watchdog.service` - Systemd unit
|
|
|
|
**Deployment**:
|
|
```bash
|
|
# On each docker LXC (docker-1, docker-2, docker-3):
|
|
sudo cp docker-prune-watchdog.sh /usr/local/bin/
|
|
sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh
|
|
sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env
|
|
sudo cp docker-prune-watchdog.service /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable docker-prune-watchdog.service
|
|
sudo systemctl start docker-prune-watchdog.service
|
|
```
|
|
|
|
**Configuration**:
|
|
Edit `/etc/docker-prune-watchdog.env`:
|
|
```ini
|
|
# Disk usage threshold (%) that triggers prune
|
|
DISK_THRESHOLD=85
|
|
|
|
# Check interval in seconds
|
|
CHECK_INTERVAL=60
|
|
|
|
# Notification mode: always, errors_only, both
|
|
NOTIFICATION_MODE=both
|
|
|
|
# Microsoft Teams webhook URL (for notifications)
|
|
TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/...
|
|
```
|
|
|
|
**Verification**:
|
|
```bash
|
|
# Check status
|
|
systemctl status docker-prune-watchdog.service
|
|
|
|
# View logs
|
|
cat /dev/shm/docker-prune-watchdog.log
|
|
tail -f /dev/shm/docker-prune-watchdog.log
|
|
|
|
# Manual check
|
|
df -h /volume1/docker-root
|
|
```
|
|
|
|
**Related Documentation**: ADR-011 (Observability & Self-Healing)
|
|
|
|
---
|
|
|
|
### 2. Proxmox Backup Hook
|
|
|
|
**Purpose**: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption
|
|
|
|
**File**: `scripts/backup/vzdump-docker-hook.sh`
|
|
|
|
**Installation**:
|
|
```bash
|
|
# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3):
|
|
sudo cp vzdump-docker-hook.sh /usr/local/bin/
|
|
sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh
|
|
```
|
|
|
|
**Configuration**:
|
|
In `/etc/vzdump.conf` or per backup job, add:
|
|
```
|
|
script: /usr/local/bin/vzdump-docker-hook.sh
|
|
```
|
|
|
|
**What it does**:
|
|
1. **On backup-start**:
|
|
- Drains Docker Swarm node
|
|
- Gracefully stops Docker service
|
|
- Sends Teams notification
|
|
|
|
2. **On post-restart**:
|
|
- Restarts Docker service
|
|
- Re-activates Swarm node
|
|
- Sends Teams notification
|
|
|
|
3. **On backup-abort**:
|
|
- Immediately restarts Docker
|
|
- Restores services
|
|
- Sends warning notification
|
|
|
|
**Docker LXC IDs**: `4031` (docker-1), `4032` (docker-2), `4033` (docker-3)
|
|
|
|
**Verification**:
|
|
```bash
|
|
# Check hook logs on Proxmox node
|
|
tail -f /var/log/vzdump-docker-hook.log
|
|
|
|
# Monitor backup job
|
|
# (Check Proxmox backup status in UI)
|
|
```
|
|
|
|
**Known Bugs (Fixed in April 2026)**:
|
|
- Unbound variable `$2` when Proxmox calls with `job-init` phase
|
|
- Fix: Changed `VMID="$2"` to `VMID="${2:-}"`
|
|
|
|
**Related Documentation**: ADR-009 (Backups)
|
|
|
|
---
|
|
|
|
## Network Configuration
|
|
|
|
### 3. MTU 9000 Jumbo Frames Migration
|
|
|
|
**Purpose**: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput
|
|
|
|
**Files**:
|
|
- `scripts/network/mtu_phase1.sh` - Write daemon.json, restart Docker
|
|
- `scripts/network/mtu_phase2.sh` - Recreate overlay networks, redeploy stacks
|
|
- `scripts/network/mtu_rollback.sh` - Restore original configuration
|
|
|
|
**Prerequisites**:
|
|
- Proxmox host MTU already set to 9000
|
|
- LXC net0 config already has `mtu=9000`
|
|
- All scripts placed in `/volume1/docker/mtu_migration/` on each docker LXC
|
|
|
|
**Deployment Steps**:
|
|
|
|
1. **Copy scripts to CephFS** (accessible by all 3 docker LXCs):
|
|
```bash
|
|
# From any docker LXC:
|
|
mkdir -p /volume1/docker/mtu_migration
|
|
cp mtu_phase1.sh /volume1/docker/mtu_migration/
|
|
cp mtu_phase2.sh /volume1/docker/mtu_migration/
|
|
cp mtu_rollback.sh /volume1/docker/mtu_migration/
|
|
chmod +x /volume1/docker/mtu_migration/mtu_*.sh
|
|
```
|
|
|
|
2. **Phase 1** (synchronized on all nodes):
|
|
```bash
|
|
# Calculate sync epoch (T+3min from now, e.g., 1755423180)
|
|
SYNC_EPOCH=$(( $(date +%s) + 180 ))
|
|
|
|
# On docker-1:
|
|
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH &
|
|
|
|
# On docker-2:
|
|
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH &
|
|
|
|
# On docker-3:
|
|
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH &
|
|
```
|
|
|
|
3. **Phase 2** (stack redeployment):
|
|
```bash
|
|
# On docker-2 (or any node with swarm leadership):
|
|
nohup /volume1/docker/mtu_migration/mtu_phase2.sh &
|
|
tail -f /volume1/docker/mtu_migration/phase2.log
|
|
```
|
|
|
|
**Monitoring Progress**:
|
|
```bash
|
|
# Check phase1 completion:
|
|
ls /volume1/docker/mtu_migration/*_phase1.done
|
|
|
|
# View logs:
|
|
cat /volume1/docker/mtu_migration/docker-1_phase1.log
|
|
cat /volume1/docker/mtu_migration/docker-2_phase1.log
|
|
cat /volume1/docker/mtu_migration/docker-3_phase1.log
|
|
|
|
# View phase2 progress:
|
|
tail -f /volume1/docker/mtu_migration/phase2.log
|
|
|
|
# Verify MTU on running interface:
|
|
ip link show eth0 | grep mtu
|
|
|
|
# Verify overlay network MTUs:
|
|
docker network ls --filter driver=overlay -q | \
|
|
xargs docker network inspect --format '{{.Name}} {{json .Options}}'
|
|
```
|
|
|
|
**Rollback**:
|
|
```bash
|
|
# On each docker LXC:
|
|
/volume1/docker/mtu_migration/mtu_rollback.sh
|
|
|
|
# Then redeploy stacks:
|
|
cd /volume1/docker/compose-files
|
|
docker stack deploy -c traefik.yaml traefik
|
|
docker stack deploy -c auth.yaml auth
|
|
docker stack deploy -c postgresql.yaml postgresql
|
|
```
|
|
|
|
**Related Documentation**: ADR-006 (Jumbo Frames)
|
|
|
|
---
|
|
|
|
## Deployment & Fixes
|
|
|
|
### 4. rc.local Boot Shebang Fix
|
|
|
|
**File**: `scripts/fixes/rc.local.fix.sh`
|
|
|
|
**Issue**: `/etc/rc.local` missing `#!/bin/bash` shebang causes boot errors
|
|
|
|
**Fix** (on docker-3):
|
|
```bash
|
|
# Add shebang to first line
|
|
sed -i '1i#!/bin/bash' /etc/rc.local
|
|
chmod +x /etc/rc.local
|
|
|
|
# Verify:
|
|
head -1 /etc/rc.local
|
|
```
|
|
|
|
---
|
|
|
|
### 5. systemd-networkd-wait-online Service Fix
|
|
|
|
**Issue**: Service waits indefinitely for all network interfaces, causes LXC boot delays
|
|
|
|
**Fix** (on docker-3):
|
|
```bash
|
|
# Mask the service
|
|
systemctl mask systemd-networkd-wait-online.service
|
|
systemctl daemon-reload
|
|
|
|
# Verify:
|
|
systemctl is-enabled systemd-networkd-wait-online.service
|
|
# Should show: masked
|
|
```
|
|
|
|
---
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
scripts/
|
|
├── monitoring/
|
|
│ ├── docker-prune-watchdog.sh
|
|
│ ├── docker-prune-watchdog.env.example
|
|
│ └── docker-prune-watchdog.service
|
|
├── backup/
|
|
│ └── vzdump-docker-hook.sh
|
|
├── network/
|
|
│ ├── mtu_phase1.sh
|
|
│ ├── mtu_phase2.sh
|
|
│ └── mtu_rollback.sh
|
|
├── fixes/
|
|
│ └── rc.local.fix.sh
|
|
└── README.md (this file)
|
|
```
|
|
|
|
---
|
|
|
|
## Related Architecture Decision Records (ADRs)
|
|
|
|
- **ADR-006**: Jumbo Frames (MTU 9000) for Docker Swarm
|
|
- **ADR-009**: Backup strategy and hook scripts
|
|
- **ADR-011**: Observability & Self-Healing (prune watchdog, Uptime Kuma)
|
|
|
|
---
|
|
|
|
## Key Notes
|
|
|
|
### Teams Webhook Deprecation
|
|
The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to:
|
|
- Power Automate flow (HTTP trigger → Post to Teams channel)
|
|
- Update all webhook URLs accordingly
|
|
|
|
### Disk Prune Aggressiveness
|
|
`docker system prune -f` is aggressive:
|
|
- Removes all stopped containers ✓
|
|
- Removes unused networks ✓
|
|
- Removes dangling images ✓
|
|
- Removes build cache ✓
|
|
- Does NOT remove volumes (safe) ✓
|
|
- Removes non-running services' images (can cause re-pulls)
|
|
|
|
**Mitigation**: Pin critical image versions in compose files (avoid `latest` tags)
|
|
|
|
### CephFS Shared Scripts
|
|
All MTU migration scripts are on shared CephFS at `/volume1/docker/mtu_migration/`, visible to all 3 docker LXCs.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Prune watchdog not triggering
|
|
```bash
|
|
# Check configuration
|
|
cat /etc/docker-prune-watchdog.env
|
|
|
|
# Verify service is running
|
|
systemctl status docker-prune-watchdog.service
|
|
|
|
# Check actual disk usage
|
|
df -h /volume1/docker-root
|
|
|
|
# Manually trigger prune if over threshold
|
|
docker system prune -f
|
|
```
|
|
|
|
### Backup hook errors
|
|
```bash
|
|
# Verify hook is installed
|
|
ls -la /usr/local/bin/vzdump-docker-hook.sh
|
|
|
|
# Check logs
|
|
tail -f /var/log/vzdump-docker-hook.log
|
|
|
|
# Test manually
|
|
/usr/local/bin/vzdump-docker-hook.sh job-init
|
|
|
|
# Verify Docker can be stopped/started
|
|
systemctl stop docker
|
|
systemctl start docker
|
|
```
|
|
|
|
### MTU migration stuck
|
|
```bash
|
|
# Check what phase completed
|
|
ls /volume1/docker/mtu_migration/*.done
|
|
|
|
# View logs
|
|
cat /volume1/docker/mtu_migration/docker-X_phase1.log
|
|
cat /volume1/docker/mtu_migration/phase2.log
|
|
|
|
# Check swarm status
|
|
docker node ls
|
|
|
|
# Verify interfaces
|
|
ip link show eth0
|
|
ip link show vmbr0
|
|
|
|
# If needed, manually complete remaining phases
|
|
# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook
|
|
```
|
|
|
|
---
|
|
|
|
## Contact & Support
|
|
|
|
For questions about these scripts, refer to:
|
|
- Implementation notes in chat history
|
|
- Architecture Decision Records (ADRs)
|
|
- Proxmox & MCP Setup documentation
|