Archived
Remove scripts documentation - moving to dedicated homelab-scripts repo
This commit is contained in:
@@ -1,359 +0,0 @@
|
|||||||
# Homelab Operational Scripts
|
|
||||||
|
|
||||||
This directory contains critical operational scripts for managing the Proxmox cluster, Docker Swarm, and homelab infrastructure.
|
|
||||||
|
|
||||||
## Quick Navigation
|
|
||||||
|
|
||||||
- **[Monitoring & Maintenance](#monitoring--maintenance)** - Docker prune watchdog, backup hooks
|
|
||||||
- **[Network Configuration](#network-configuration)** - MTU migration for Jumbo Frames
|
|
||||||
- **[Deployment & Fixes](#deployment--fixes)** - Boot time fixes, systemd configuration
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Monitoring & Maintenance
|
|
||||||
|
|
||||||
### 1. Docker Prune Watchdog
|
|
||||||
|
|
||||||
**Purpose**: Automatically manages disk space on Docker nodes to prevent "disk full" failures
|
|
||||||
|
|
||||||
**Files**:
|
|
||||||
- `scripts/monitoring/docker-prune-watchdog.sh` - Main watchdog script
|
|
||||||
- `scripts/monitoring/docker-prune-watchdog.env.example` - Configuration template
|
|
||||||
- `scripts/monitoring/docker-prune-watchdog.service` - Systemd unit
|
|
||||||
|
|
||||||
**Deployment**:
|
|
||||||
```bash
|
|
||||||
# On each docker LXC (docker-1, docker-2, docker-3):
|
|
||||||
sudo cp docker-prune-watchdog.sh /usr/local/bin/
|
|
||||||
sudo chmod +x /usr/local/bin/docker-prune-watchdog.sh
|
|
||||||
sudo cp docker-prune-watchdog.env /etc/docker-prune-watchdog.env
|
|
||||||
sudo cp docker-prune-watchdog.service /etc/systemd/system/
|
|
||||||
sudo systemctl daemon-reload
|
|
||||||
sudo systemctl enable docker-prune-watchdog.service
|
|
||||||
sudo systemctl start docker-prune-watchdog.service
|
|
||||||
```
|
|
||||||
|
|
||||||
**Configuration**:
|
|
||||||
Edit `/etc/docker-prune-watchdog.env`:
|
|
||||||
```ini
|
|
||||||
# Disk usage threshold (%) that triggers prune
|
|
||||||
DISK_THRESHOLD=85
|
|
||||||
|
|
||||||
# Check interval in seconds
|
|
||||||
CHECK_INTERVAL=60
|
|
||||||
|
|
||||||
# Notification mode: always, errors_only, both
|
|
||||||
NOTIFICATION_MODE=both
|
|
||||||
|
|
||||||
# Microsoft Teams webhook URL (for notifications)
|
|
||||||
TEAMS_WEBHOOK_URL=https://bryanfamily.webhook.office.com/webhookb2/...
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verification**:
|
|
||||||
```bash
|
|
||||||
# Check status
|
|
||||||
systemctl status docker-prune-watchdog.service
|
|
||||||
|
|
||||||
# View logs
|
|
||||||
cat /dev/shm/docker-prune-watchdog.log
|
|
||||||
tail -f /dev/shm/docker-prune-watchdog.log
|
|
||||||
|
|
||||||
# Manual check
|
|
||||||
df -h /volume1/docker-root
|
|
||||||
```
|
|
||||||
|
|
||||||
**Related Documentation**: ADR-011 (Observability & Self-Healing)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2. Proxmox Backup Hook
|
|
||||||
|
|
||||||
**Purpose**: Gracefully stops/restarts Docker containers around Proxmox backup snapshots to prevent overlay network corruption
|
|
||||||
|
|
||||||
**File**: `scripts/backup/vzdump-docker-hook.sh`
|
|
||||||
|
|
||||||
**Installation**:
|
|
||||||
```bash
|
|
||||||
# On EACH Proxmox node (nuck7-1, nuck7-2, nuck7-3):
|
|
||||||
sudo cp vzdump-docker-hook.sh /usr/local/bin/
|
|
||||||
sudo chmod +x /usr/local/bin/vzdump-docker-hook.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
**Configuration**:
|
|
||||||
In `/etc/vzdump.conf` or per backup job, add:
|
|
||||||
```
|
|
||||||
script: /usr/local/bin/vzdump-docker-hook.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does**:
|
|
||||||
1. **On backup-start**:
|
|
||||||
- Drains Docker Swarm node
|
|
||||||
- Gracefully stops Docker service
|
|
||||||
- Sends Teams notification
|
|
||||||
|
|
||||||
2. **On post-restart**:
|
|
||||||
- Restarts Docker service
|
|
||||||
- Re-activates Swarm node
|
|
||||||
- Sends Teams notification
|
|
||||||
|
|
||||||
3. **On backup-abort**:
|
|
||||||
- Immediately restarts Docker
|
|
||||||
- Restores services
|
|
||||||
- Sends warning notification
|
|
||||||
|
|
||||||
**Docker LXC IDs**: `4031` (docker-1), `4032` (docker-2), `4033` (docker-3)
|
|
||||||
|
|
||||||
**Verification**:
|
|
||||||
```bash
|
|
||||||
# Check hook logs on Proxmox node
|
|
||||||
tail -f /var/log/vzdump-docker-hook.log
|
|
||||||
|
|
||||||
# Monitor backup job
|
|
||||||
# (Check Proxmox backup status in UI)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Known Bugs (Fixed in April 2026)**:
|
|
||||||
- Unbound variable `$2` when Proxmox calls with `job-init` phase
|
|
||||||
- Fix: Changed `VMID="$2"` to `VMID="${2:-}"`
|
|
||||||
|
|
||||||
**Related Documentation**: ADR-009 (Backups)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Network Configuration
|
|
||||||
|
|
||||||
### 3. MTU 9000 Jumbo Frames Migration
|
|
||||||
|
|
||||||
**Purpose**: Migrate Docker Swarm from standard MTU 1500 to Jumbo Frames (MTU 9000) for improved throughput
|
|
||||||
|
|
||||||
**Files**:
|
|
||||||
- `scripts/network/mtu_phase1.sh` - Write daemon.json, restart Docker
|
|
||||||
- `scripts/network/mtu_phase2.sh` - Recreate overlay networks, redeploy stacks
|
|
||||||
- `scripts/network/mtu_rollback.sh` - Restore original configuration
|
|
||||||
|
|
||||||
**Prerequisites**:
|
|
||||||
- Proxmox host MTU already set to 9000
|
|
||||||
- LXC net0 config already has `mtu=9000`
|
|
||||||
- All scripts placed in `/volume1/docker/mtu_migration/` on each docker LXC
|
|
||||||
|
|
||||||
**Deployment Steps**:
|
|
||||||
|
|
||||||
1. **Copy scripts to CephFS** (accessible by all 3 docker LXCs):
|
|
||||||
```bash
|
|
||||||
# From any docker LXC:
|
|
||||||
mkdir -p /volume1/docker/mtu_migration
|
|
||||||
cp mtu_phase1.sh /volume1/docker/mtu_migration/
|
|
||||||
cp mtu_phase2.sh /volume1/docker/mtu_migration/
|
|
||||||
cp mtu_rollback.sh /volume1/docker/mtu_migration/
|
|
||||||
chmod +x /volume1/docker/mtu_migration/mtu_*.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Phase 1** (synchronized on all nodes):
|
|
||||||
```bash
|
|
||||||
# Calculate sync epoch (T+3min from now, e.g., 1755423180)
|
|
||||||
SYNC_EPOCH=$(( $(date +%s) + 180 ))
|
|
||||||
|
|
||||||
# On docker-1:
|
|
||||||
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-1 $SYNC_EPOCH &
|
|
||||||
|
|
||||||
# On docker-2:
|
|
||||||
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-2 $SYNC_EPOCH &
|
|
||||||
|
|
||||||
# On docker-3:
|
|
||||||
nohup /volume1/docker/mtu_migration/mtu_phase1.sh docker-3 $SYNC_EPOCH &
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Phase 2** (stack redeployment):
|
|
||||||
```bash
|
|
||||||
# On docker-2 (or any node with swarm leadership):
|
|
||||||
nohup /volume1/docker/mtu_migration/mtu_phase2.sh &
|
|
||||||
tail -f /volume1/docker/mtu_migration/phase2.log
|
|
||||||
```
|
|
||||||
|
|
||||||
**Monitoring Progress**:
|
|
||||||
```bash
|
|
||||||
# Check phase1 completion:
|
|
||||||
ls /volume1/docker/mtu_migration/*_phase1.done
|
|
||||||
|
|
||||||
# View logs:
|
|
||||||
cat /volume1/docker/mtu_migration/docker-1_phase1.log
|
|
||||||
cat /volume1/docker/mtu_migration/docker-2_phase1.log
|
|
||||||
cat /volume1/docker/mtu_migration/docker-3_phase1.log
|
|
||||||
|
|
||||||
# View phase2 progress:
|
|
||||||
tail -f /volume1/docker/mtu_migration/phase2.log
|
|
||||||
|
|
||||||
# Verify MTU on running interface:
|
|
||||||
ip link show eth0 | grep mtu
|
|
||||||
|
|
||||||
# Verify overlay network MTUs:
|
|
||||||
docker network ls --filter driver=overlay -q | \
|
|
||||||
xargs docker network inspect --format '{{.Name}} {{json .Options}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rollback**:
|
|
||||||
```bash
|
|
||||||
# On each docker LXC:
|
|
||||||
/volume1/docker/mtu_migration/mtu_rollback.sh
|
|
||||||
|
|
||||||
# Then redeploy stacks:
|
|
||||||
cd /volume1/docker/compose-files
|
|
||||||
docker stack deploy -c traefik.yaml traefik
|
|
||||||
docker stack deploy -c auth.yaml auth
|
|
||||||
docker stack deploy -c postgresql.yaml postgresql
|
|
||||||
```
|
|
||||||
|
|
||||||
**Related Documentation**: ADR-006 (Jumbo Frames)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Deployment & Fixes
|
|
||||||
|
|
||||||
### 4. rc.local Boot Shebang Fix
|
|
||||||
|
|
||||||
**File**: `scripts/fixes/rc.local.fix.sh`
|
|
||||||
|
|
||||||
**Issue**: `/etc/rc.local` missing `#!/bin/bash` shebang causes boot errors
|
|
||||||
|
|
||||||
**Fix** (on docker-3):
|
|
||||||
```bash
|
|
||||||
# Add shebang to first line
|
|
||||||
sed -i '1i#!/bin/bash' /etc/rc.local
|
|
||||||
chmod +x /etc/rc.local
|
|
||||||
|
|
||||||
# Verify:
|
|
||||||
head -1 /etc/rc.local
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 5. systemd-networkd-wait-online Service Fix
|
|
||||||
|
|
||||||
**Issue**: Service waits indefinitely for all network interfaces, causes LXC boot delays
|
|
||||||
|
|
||||||
**Fix** (on docker-3):
|
|
||||||
```bash
|
|
||||||
# Mask the service
|
|
||||||
systemctl mask systemd-networkd-wait-online.service
|
|
||||||
systemctl daemon-reload
|
|
||||||
|
|
||||||
# Verify:
|
|
||||||
systemctl is-enabled systemd-networkd-wait-online.service
|
|
||||||
# Should show: masked
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Directory Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
scripts/
|
|
||||||
├── monitoring/
|
|
||||||
│ ├── docker-prune-watchdog.sh
|
|
||||||
│ ├── docker-prune-watchdog.env.example
|
|
||||||
│ └── docker-prune-watchdog.service
|
|
||||||
├── backup/
|
|
||||||
│ └── vzdump-docker-hook.sh
|
|
||||||
├── network/
|
|
||||||
│ ├── mtu_phase1.sh
|
|
||||||
│ ├── mtu_phase2.sh
|
|
||||||
│ └── mtu_rollback.sh
|
|
||||||
├── fixes/
|
|
||||||
│ └── rc.local.fix.sh
|
|
||||||
└── README.md (this file)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Related Architecture Decision Records (ADRs)
|
|
||||||
|
|
||||||
- **ADR-006**: Jumbo Frames (MTU 9000) for Docker Swarm
|
|
||||||
- **ADR-009**: Backup strategy and hook scripts
|
|
||||||
- **ADR-011**: Observability & Self-Healing (prune watchdog, Uptime Kuma)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Key Notes
|
|
||||||
|
|
||||||
### Teams Webhook Deprecation
|
|
||||||
The Office 365 Connector webhook format is deprecated by Microsoft. Monitor for outages and plan migration to:
|
|
||||||
- Power Automate flow (HTTP trigger → Post to Teams channel)
|
|
||||||
- Update all webhook URLs accordingly
|
|
||||||
|
|
||||||
### Disk Prune Aggressiveness
|
|
||||||
`docker system prune -f` is aggressive:
|
|
||||||
- Removes all stopped containers ✓
|
|
||||||
- Removes unused networks ✓
|
|
||||||
- Removes dangling images ✓
|
|
||||||
- Removes build cache ✓
|
|
||||||
- Does NOT remove volumes (safe) ✓
|
|
||||||
- Removes non-running services' images (can cause re-pulls)
|
|
||||||
|
|
||||||
**Mitigation**: Pin critical image versions in compose files (avoid `latest` tags)
|
|
||||||
|
|
||||||
### CephFS Shared Scripts
|
|
||||||
All MTU migration scripts are on shared CephFS at `/volume1/docker/mtu_migration/`, visible to all 3 docker LXCs.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Prune watchdog not triggering
|
|
||||||
```bash
|
|
||||||
# Check configuration
|
|
||||||
cat /etc/docker-prune-watchdog.env
|
|
||||||
|
|
||||||
# Verify service is running
|
|
||||||
systemctl status docker-prune-watchdog.service
|
|
||||||
|
|
||||||
# Check actual disk usage
|
|
||||||
df -h /volume1/docker-root
|
|
||||||
|
|
||||||
# Manually trigger prune if over threshold
|
|
||||||
docker system prune -f
|
|
||||||
```
|
|
||||||
|
|
||||||
### Backup hook errors
|
|
||||||
```bash
|
|
||||||
# Verify hook is installed
|
|
||||||
ls -la /usr/local/bin/vzdump-docker-hook.sh
|
|
||||||
|
|
||||||
# Check logs
|
|
||||||
tail -f /var/log/vzdump-docker-hook.log
|
|
||||||
|
|
||||||
# Test manually
|
|
||||||
/usr/local/bin/vzdump-docker-hook.sh job-init
|
|
||||||
|
|
||||||
# Verify Docker can be stopped/started
|
|
||||||
systemctl stop docker
|
|
||||||
systemctl start docker
|
|
||||||
```
|
|
||||||
|
|
||||||
### MTU migration stuck
|
|
||||||
```bash
|
|
||||||
# Check what phase completed
|
|
||||||
ls /volume1/docker/mtu_migration/*.done
|
|
||||||
|
|
||||||
# View logs
|
|
||||||
cat /volume1/docker/mtu_migration/docker-X_phase1.log
|
|
||||||
cat /volume1/docker/mtu_migration/phase2.log
|
|
||||||
|
|
||||||
# Check swarm status
|
|
||||||
docker node ls
|
|
||||||
|
|
||||||
# Verify interfaces
|
|
||||||
ip link show eth0
|
|
||||||
ip link show vmbr0
|
|
||||||
|
|
||||||
# If needed, manually complete remaining phases
|
|
||||||
# See "Troubleshooting" section in MTU 9000 Jumbo Frames Migration Runbook
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Contact & Support
|
|
||||||
|
|
||||||
For questions about these scripts, refer to:
|
|
||||||
- Implementation notes in chat history
|
|
||||||
- Architecture Decision Records (ADRs)
|
|
||||||
- Proxmox & MCP Setup documentation
|
|
||||||
Reference in New Issue
Block a user