Archived
Add comprehensive scripts documentation and operational reference
This commit is contained in:
@@ -0,0 +1,359 @@
|
||||
# Homelab Operational Scripts
|
||||
|
||||
This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure.
|
||||
|
||||
## Quick Navigation
|
||||
|
||||
- **[Monitoring & Maintenance](#monitoring--maintenance)** - Docker prune watchdog, backup hooks
|
||||
- **[Network Configuration](#network-configuration)** - MTU migration for Jumbo Frames
|
||||
- **[Deployment & Fixes](#deployment--fixes)** - Boot time fixes, systemd configuration
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Maintenance
|
||||
|
||||
### 1. Docker Prune Watchdog
|
||||
|
||||
**Purpose**: Automatically manages disk space on Docker nodes to prevent "disk full" failures
|
||||
|
||||
**Files**:
|
||||
- `scripts/monitoring/docker-prune-watchdog.sh` - Main watchdog script
|
||||
- `scripts/monitoring/docker-prune-watchdog.env.example` - Configuration template
|
||||
- `scripts/monitoring/docker-prune-watchdog.service` - Systemd unit
|
||||
|
||||
**Deployment**:
|
||||
```bash
|
||||
# On each docker LXC (docker-1, docker-2, docker-3):
|
||||
sudo cp docker-prune-watchdog.sh /usr/local/bin/
|
||||
sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh
|
||||
sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env
|
||||
sudo cp docker-prune-watchdog.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable docker-prune-watchdog.service
|
||||
sudo systemctl start docker-prune-watchdog.service
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
Edit `/etc/docker-prune-watchdog.env`:
|
||||
```ini
|
||||
# Disk usage threshold (%) that triggers prune
|
||||
DISK_THRESHOLD=85
|
||||
|
||||
# Check interval in seconds
|
||||
CHECK_INTERVAL=60
|
||||
|
||||
# Notification mode: always, errors_only, both
|
||||
NOTIFICATION_MODE=both
|
||||
|
||||
# Microsoft Teams webhook URL (for notifications)
|
||||
TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/...
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
# Check status
|
||||
systemctl status docker-prune-watchdog.service
|
||||
|
||||
# View logs
|
||||
cat /dev/shm/docker-prune-watchdog.log
|
||||
tail -f /dev/shm/docker-prune-watchdog.log
|
||||
|
||||
# Manual check
|
||||
df -h /volume1/docker-root
|
||||
```
|
||||
|
||||
**Related Documentation**: ADR-011 (Observability & Self-Healing)
|
||||
|
||||
---
|
||||
|
||||
### 2. Proxmox Backup Hook
|
||||
|
||||
**Purpose**: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption
|
||||
|
||||
**File**: `scripts/backup/vzdump-docker-hook.sh`
|
||||
|
||||
**Installation**:
|
||||
```bash
|
||||
# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3):
|
||||
sudo cp vzdump-docker-hook.sh /usr/local/bin/
|
||||
sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
In `/etc/vzdump.conf` or per backup job, add:
|
||||
```
|
||||
script: /usr/local/bin/vzdump-docker-hook.sh
|
||||
```
|
||||
|
||||
**What it does**:
|
||||
1. **On backup-start**:
|
||||
- Drains Docker Swarm node
|
||||
- Gracefully stops Docker service
|
||||
- Sends Teams notification
|
||||
|
||||
2. **On post-restart**:
|
||||
- Restarts Docker service
|
||||
- Re-activates Swarm node
|
||||
- Sends Teams notification
|
||||
|
||||
3. **On backup-abort**:
|
||||
- Immediately restarts Docker
|
||||
- Restores services
|
||||
- Sends warning notification
|
||||
|
||||
**Docker LXC IDs**: `4031` (docker-1), `4032` (docker-2), `4033` (docker-3)
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
# Check hook logs on Proxmox node
|
||||
tail -f /var/log/vzdump-docker-hook.log
|
||||
|
||||
# Monitor backup job
|
||||
# (Check Proxmox backup status in UI)
|
||||
```
|
||||
|
||||
**Known Bugs (Fixed in April 2026)**:
|
||||
- Unbound variable `$2` when Proxmox calls with `job-init` phase
|
||||
- Fix: Changed `VMID="$2"` to `VMID="${2:-}"`
|
||||
|
||||
**Related Documentation**: ADR-009 (Backups)
|
||||
|
||||
---
|
||||
|
||||
## Network Configuration
|
||||
|
||||
### 3. MTU 9000 Jumbo Frames Migration
|
||||
|
||||
**Purpose**: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput
|
||||
|
||||
**Files**:
|
||||
- `scripts/network/mtu_phase1.sh` - Write daemon.json, restart Docker
|
||||
- `scripts/network/mtu_phase2.sh` - Recreate overlay networks, redeploy stacks
|
||||
- `scripts/network/mtu_rollback.sh` - Restore original configuration
|
||||
|
||||
**Prerequisites**:
|
||||
- Proxmox host MTU already set to 9000
|
||||
- LXC net0 config already has `mtu=9000`
|
||||
- All scripts placed in `/volume1/docker/mtu_migration/` on each docker LXC
|
||||
|
||||
**Deployment Steps**:
|
||||
|
||||
1. **Copy scripts to CephFS** (accessible by all 3 docker LXCs):
|
||||
```bash
|
||||
# From any docker LXC:
|
||||
mkdir -p /volume1/docker/mtu_migration
|
||||
cp mtu_phase1.sh /volume1/docker/mtu_migration/
|
||||
cp mtu_phase2.sh /volume1/docker/mtu_migration/
|
||||
cp mtu_rollback.sh /volume1/docker/mtu_migration/
|
||||
chmod +x /volume1/docker/mtu_migration/mtu_*.sh
|
||||
```
|
||||
|
||||
2. **Phase 1** (synchronized on all nodes):
|
||||
```bash
|
||||
# Calculate sync epoch (T+3min from now, e.g., 1755423180)
|
||||
SYNC_EPOCH=$(( $(date +%s) + 180 ))
|
||||
|
||||
# On docker-1:
|
||||
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH &
|
||||
|
||||
# On docker-2:
|
||||
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH &
|
||||
|
||||
# On docker-3:
|
||||
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH &
|
||||
```
|
||||
|
||||
3. **Phase 2** (stack redeployment):
|
||||
```bash
|
||||
# On docker-2 (or any node with swarm leadership):
|
||||
nohup /volume1/docker/mtu_migration/mtu_phase2.sh &
|
||||
tail -f /volume1/docker/mtu_migration/phase2.log
|
||||
```
|
||||
|
||||
**Monitoring Progress**:
|
||||
```bash
|
||||
# Check phase1 completion:
|
||||
ls /volume1/docker/mtu_migration/*_phase1.done
|
||||
|
||||
# View logs:
|
||||
cat /volume1/docker/mtu_migration/docker-1_phase1.log
|
||||
cat /volume1/docker/mtu_migration/docker-2_phase1.log
|
||||
cat /volume1/docker/mtu_migration/docker-3_phase1.log
|
||||
|
||||
# View phase2 progress:
|
||||
tail -f /volume1/docker/mtu_migration/phase2.log
|
||||
|
||||
# Verify MTU on running interface:
|
||||
ip link show eth0 | grep mtu
|
||||
|
||||
# Verify overlay network MTUs:
|
||||
docker network ls --filter driver=overlay -q | \
|
||||
xargs docker network inspect --format '{{.Name}} {{json .Options}}'
|
||||
```
|
||||
|
||||
**Rollback**:
|
||||
```bash
|
||||
# On each docker LXC:
|
||||
/volume1/docker/mtu_migration/mtu_rollback.sh
|
||||
|
||||
# Then redeploy stacks:
|
||||
cd /volume1/docker/compose-files
|
||||
docker stack deploy -c traefik.yaml traefik
|
||||
docker stack deploy -c auth.yaml auth
|
||||
docker stack deploy -c postgresql.yaml postgresql
|
||||
```
|
||||
|
||||
**Related Documentation**: ADR-006 (Jumbo Frames)
|
||||
|
||||
---
|
||||
|
||||
## Deployment & Fixes
|
||||
|
||||
### 4. rc.local Boot Shebang Fix
|
||||
|
||||
**File**: `scripts/fixes/rc.local.fix.sh`
|
||||
|
||||
**Issue**: `/etc/rc.local` missing `#!/bin/bash` shebang causes boot errors
|
||||
|
||||
**Fix** (on docker-3):
|
||||
```bash
|
||||
# Add shebang to first line
|
||||
sed -i '1i#!/bin/bash' /etc/rc.local
|
||||
chmod +x /etc/rc.local
|
||||
|
||||
# Verify:
|
||||
head -1 /etc/rc.local
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. systemd-networkd-wait-online Service Fix
|
||||
|
||||
**Issue**: Service waits indefinitely for all network interfaces, causes LXC boot delays
|
||||
|
||||
**Fix** (on docker-3):
|
||||
```bash
|
||||
# Mask the service
|
||||
systemctl mask systemd-networkd-wait-online.service
|
||||
systemctl daemon-reload
|
||||
|
||||
# Verify:
|
||||
systemctl is-enabled systemd-networkd-wait-online.service
|
||||
# Should show: masked
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
scripts/
|
||||
├── monitoring/
|
||||
│ ├── docker-prune-watchdog.sh
|
||||
│ ├── docker-prune-watchdog.env.example
|
||||
│ └── docker-prune-watchdog.service
|
||||
├── backup/
|
||||
│ └── vzdump-docker-hook.sh
|
||||
├── network/
|
||||
│ ├── mtu_phase1.sh
|
||||
│ ├── mtu_phase2.sh
|
||||
│ └── mtu_rollback.sh
|
||||
├── fixes/
|
||||
│ └── rc.local.fix.sh
|
||||
└── README.md (this file)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Architecture Decision Records (ADRs)
|
||||
|
||||
- **ADR-006**: Jumbo Frames (MTU 9000) for Docker Swarm
|
||||
- **ADR-009**: Backup strategy and hook scripts
|
||||
- **ADR-011**: Observability & Self-Healing (prune watchdog, Uptime Kuma)
|
||||
|
||||
---
|
||||
|
||||
## Key Notes
|
||||
|
||||
### Teams Webhook Deprecation
|
||||
The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to:
|
||||
- Power Automate flow (HTTP trigger → Post to Teams channel)
|
||||
- Update all webhook URLs accordingly
|
||||
|
||||
### Disk Prune Aggressiveness
|
||||
`docker system prune -f` is aggressive:
|
||||
- Removes all stopped containers ✓
|
||||
- Removes unused networks ✓
|
||||
- Removes dangling images ✓
|
||||
- Removes build cache ✓
|
||||
- Does NOT remove volumes (safe) ✓
|
||||
- Removes non-running services' images (can cause re-pulls)
|
||||
|
||||
**Mitigation**: Pin critical image versions in compose files (avoid `latest` tags)
|
||||
|
||||
### CephFS Shared Scripts
|
||||
All MTU migration scripts are on shared CephFS at `/volume1/docker/mtu_migration/`, visible to all 3 docker LXCs.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Prune watchdog not triggering
|
||||
```bash
|
||||
# Check configuration
|
||||
cat /etc/docker-prune-watchdog.env
|
||||
|
||||
# Verify service is running
|
||||
systemctl status docker-prune-watchdog.service
|
||||
|
||||
# Check actual disk usage
|
||||
df -h /volume1/docker-root
|
||||
|
||||
# Manually trigger prune if over threshold
|
||||
docker system prune -f
|
||||
```
|
||||
|
||||
### Backup hook errors
|
||||
```bash
|
||||
# Verify hook is installed
|
||||
ls -la /usr/local/bin/vzdump-docker-hook.sh
|
||||
|
||||
# Check logs
|
||||
tail -f /var/log/vzdump-docker-hook.log
|
||||
|
||||
# Test manually
|
||||
/usr/local/bin/vzdump-docker-hook.sh job-init
|
||||
|
||||
# Verify Docker can be stopped/started
|
||||
systemctl stop docker
|
||||
systemctl start docker
|
||||
```
|
||||
|
||||
### MTU migration stuck
|
||||
```bash
|
||||
# Check what phase completed
|
||||
ls /volume1/docker/mtu_migration/*.done
|
||||
|
||||
# View logs
|
||||
cat /volume1/docker/mtu_migration/docker-X_phase1.log
|
||||
cat /volume1/docker/mtu_migration/phase2.log
|
||||
|
||||
# Check swarm status
|
||||
docker node ls
|
||||
|
||||
# Verify interfaces
|
||||
ip link show eth0
|
||||
ip link show vmbr0
|
||||
|
||||
# If needed, manually complete remaining phases
|
||||
# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contact & Support
|
||||
|
||||
For questions about these scripts, refer to:
|
||||
- Implementation notes in chat history
|
||||
- Architecture Decision Records (ADRs)
|
||||
- Proxmox & MCP Setup documentation
|
||||
Reference in New Issue
Block a user