From 04918c2c1f62dc6a0987f8615d44c376ba89918e Mon Sep 17 00:00:00 2001 From: admin Date: Thu, 18 Jun 2026 22:45:05 -0700 Subject: [PATCH] Add main README for repository organization --- README.md | 342 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 342 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..69c20cc --- /dev/null +++ b/README.md @@ -0,0 +1,342 @@ +# Homelab Infrastructure Repository + +This Gitea repository contains all configuration files and scripts for managing the homelab infrastructure: + +- **Docker Swarm** cluster (3 nodes: docker-1, docker-2, docker-3) +- **Proxmox** cluster (3 nodes: nuck7-1, nuck7-2, nuck7-3) +- **Operational automation** (backups, monitoring, network management) + +--- + +## 📁 Directory Structure + +``` +├── compose-files/ # Docker Swarm compose files +│ ├── traefik.yaml # Reverse proxy & load balancer +│ ├── auth.yaml # Authentik authentication +│ ├── postgresql.yaml # PostgreSQL database +│ ├── ... (other stacks) +│ └── (add new services here) +│ +├── scripts/ # Operational scripts +│ ├── monitoring/ # Disk & service monitoring +│ │ ├── docker-prune-watchdog.sh +│ │ ├── docker-prune-watchdog.env.example +│ │ └── docker-prune-watchdog.service +│ ├── backup/ # Backup automation +│ │ └── vzdump-docker-hook.sh +│ ├── network/ # Network configuration +│ │ ├── mtu_phase1.sh +│ │ ├── mtu_phase2.sh +│ │ └── mtu_rollback.sh +│ └── fixes/ # One-time boot fixes +│ └── rc.local.fix.sh +│ +├── SCRIPTS-README.md # Detailed scripts documentation +├── README.md # This file +└── .gitignore # Don't commit secrets, logs, etc. + +``` + +--- + +## 🚀 Quick Start + +### Using Docker Swarm Compose Files + +```bash +# SSH into a Docker LXC +ssh root@docker-1 # or docker-2, docker-3 + +# Clone the repo locally +cd /volume1/docker +git clone https://git.bryanmail.net/admin/compose-files.git repo +cd repo + +# Deploy a stack +docker stack deploy -c traefik.yaml traefik +docker stack deploy -c auth.yaml auth +docker stack deploy -c postgresql.yaml postgresql + +# View running stacks +docker stack ls +docker service ls +``` + +### Managing & Updating Stacks + +```bash +# Pull latest from repo +git pull origin main + +# Update a specific stack (re-deploy) +docker stack deploy -c traefik.yaml traefik + +# Remove a stack +docker stack rm traefik + +# View stack status +docker stack ps traefik + +# View service logs +docker service logs traefik_reverse-proxy +``` + +--- + +## 🛠️ Operational Scripts + +### **[→ Full Scripts Documentation](SCRIPTS-README.md)** + +Quick reference: + +#### 1. **Docker Prune Watchdog** +Automatically manages disk space on Docker nodes. +```bash +# Check status +systemctl status docker-prune-watchdog.service + +# View logs +tail -f /dev/shm/docker-prune-watchdog.log + +# Configuration +cat /etc/docker-prune-watchdog.env +``` + +#### 2. **Proxmox Backup Hook** +Gracefully stops Docker during backups to prevent corruption. +```bash +# Installed at +/usr/local/bin/vzdump-docker-hook.sh + +# Check logs +tail -f /var/log/vzdump-docker-hook.log +``` + +#### 3. **MTU 9000 Migration** +Migrate Docker Swarm to Jumbo Frames for improved performance. +```bash +# See SCRIPTS-README.md for detailed steps +``` + +--- + +## 📋 Architecture + +### Proxmox Cluster +``` +nuck7-1 (Hypervisor) nuck7-2 (Hypervisor) nuck7-3 (Hypervisor) +├─ docker-1 (LXC 4031) ├─ docker-2 (LXC 4032) ├─ docker-3 (LXC 4033) +│ └─ Docker Swarm Node │ └─ Docker Swarm Node │ └─ Docker Swarm Node +│ (Manager) │ (Leader) │ (Manager) +└─ ...other LXCs └─ ...other LXCs └─ ...other LXCs +``` + +### Storage +- **CephFS** mounted at `/volume1/docker/` (shared across all docker LXCs) +- **Compose files**: `/volume1/docker/compose-files/` +- **Scripts**: `/volume1/docker/mtu_migration/`, `/usr/local/bin/`, etc. +- **Data volumes**: PostgreSQL, Authentik, and other services use named volumes or `/volume1/docker/` mounts + +### Networking +- **VIP**: 192.168.4.30 (Keepalived, managed by docker-2) +- **Docker hosts**: 192.168.4.31 (docker-1), 192.168.4.32 (docker-2), 192.168.4.33 (docker-3) +- **Traefik**: Reverse proxy with Let's Encrypt TLS + Authentik forward auth +- **Domain**: bryanmail.net + +--- + +## 🔧 Common Tasks + +### Deploy a New Service + +1. **Create compose file** in `/volume1/docker/compose-files/myservice.yaml` +2. **Test locally**: + ```bash + docker compose -f myservice.yaml up -d + docker compose -f myservice.yaml logs + ``` +3. **Convert to Swarm** (remove `container_name`, use `services:` format for Swarm) +4. **Deploy to Swarm**: + ```bash + docker stack deploy -c myservice.yaml myservice + ``` +5. **Commit to repo**: + ```bash + git add myservice.yaml + git commit -m "Add myservice stack" + git push origin main + ``` + +### Update a Running Service + +```bash +# Edit the compose file +nano /volume1/docker/compose-files/traefik.yaml + +# Redeploy +docker stack deploy -c /volume1/docker/compose-files/traefik.yaml traefik + +# Verify changes +docker stack ps traefik +``` + +### Check Disk Space + +```bash +# On docker host +df -h /volume1/docker-root + +# View prune watchdog status +systemctl status docker-prune-watchdog.service +tail -f /dev/shm/docker-prune-watchdog.log +``` + +### Monitor Services + +```bash +# List all services +docker service ls + +# Get service details +docker service inspect traefik_reverse-proxy + +# View service logs +docker service logs -f traefik_reverse-proxy + +# Check service tasks (containers) +docker service ps traefik_reverse-proxy +``` + +--- + +## ⚠️ Important Notes + +### Secrets Management +**DO NOT commit secrets, passwords, or API keys to this repo.** + +Use one of these approaches: +1. **Docker Secrets** (for Swarm) + ```yaml + services: + myapp: + secrets: + - db_password + secrets: + db_password: + external: true + ``` + +2. **Environment files** (add to `.gitignore`) + ```bash + # Create .env (not tracked by git) + echo "DB_PASSWORD=mysecretpassword" > .env + + # Use in compose + env_file: .env + ``` + +3. **Docker Secrets CLI** + ```bash + echo "mysecretpassword" | docker secret create db_password - + ``` + +### Git Workflow + +```bash +# Before starting work +git pull origin main + +# Create a feature branch for new services +git checkout -b feature/new-service + +# Make changes +# ... edit files ... + +# Commit +git add . +git commit -m "Add new-service stack with documentation" + +# Push +git push origin feature/new-service + +# Create PR for review (if using pull requests) +``` + +### Backup Strategy +- **Proxmox backups**: Automatic via vzdump-docker-hook.sh +- **Database exports**: Automated via Cronicle jobs +- **Configurations**: Version controlled in this repo (compose files, scripts) +- **Volumes**: Proxmox handles via CephFS snapshots + +--- + +## 📚 Related Documentation + +- **Architecture Decision Records (ADRs)**: See notes + - ADR-006: Jumbo Frames (MTU 9000) + - ADR-009: Backup Strategy + - ADR-011: Observability & Self-Healing + +- **Detailed Scripts**: [→ SCRIPTS-README.md](SCRIPTS-README.md) +- **Proxmox Setup**: Documented in separate notes +- **MCP Servers**: Proxmox MCP configured in Open WebUI + +--- + +## 🐛 Troubleshooting + +### Stack won't deploy +```bash +# Check syntax +docker-compose config -f myservice.yaml + +# Check node availability +docker node ls + +# Check disk space +df -h /volume1/docker-root + +# Check Docker status +systemctl status docker +``` + +### Service container keeps restarting +```bash +# View service logs +docker service logs -f myservice_container + +# Inspect container state +docker ps -a | grep myservice +``` + +### Network issues +```bash +# Check overlay networks +docker network ls --filter driver=overlay + +# Verify network connectivity +docker run --rm --network traefik_backend alpine ping traefik_reverse-proxy +``` + +--- + +## 🤝 Contributing + +When adding new services: + +1. **Use Swarm-compatible YAML** (no `container_name`, proper `services:` format) +2. **Document requirements** in compose file comments +3. **Test on non-production first** (docker-compose up on single host) +4. **Add notes** about persistent volumes, secrets, networking +5. **Update this README** if adding infrastructure changes + +--- + +## 📞 Support + +For detailed script documentation: [→ SCRIPTS-README.md](SCRIPTS-README.md) + +For architecture decisions: See ADRs in notes + +For troubleshooting: Check logs, verify disk space, check Docker Swarm status