diff --git a/README.md b/README.md index 69c20cc..cef9dbe 100644 --- a/README.md +++ b/README.md @@ -1,54 +1,33 @@ -# Homelab Infrastructure Repository +# Homelab Infrastructure Repository - Docker Swarm Compose Files -This Gitea repository contains all configuration files and scripts for managing the homelab infrastructure: +This Gitea repository contains **Docker Swarm compose files** for all services running in the homelab. -- **Docker Swarm** cluster (3 nodes: docker-1, docker-2, docker-3) -- **Proxmox** cluster (3 nodes: nuck7-1, nuck7-2, nuck7-3) -- **Operational automation** (backups, monitoring, network management) +**For operational scripts** (backup hooks, prune watchdog, network configuration), see the separate **`homelab-scripts`** repository. --- ## 📁 Directory Structure ``` -├── compose-files/ # Docker Swarm compose files -│ ├── traefik.yaml # Reverse proxy & load balancer -│ ├── auth.yaml # Authentik authentication -│ ├── postgresql.yaml # PostgreSQL database -│ ├── ... (other stacks) -│ └── (add new services here) -│ -├── scripts/ # Operational scripts -│ ├── monitoring/ # Disk & service monitoring -│ │ ├── docker-prune-watchdog.sh -│ │ ├── docker-prune-watchdog.env.example -│ │ └── docker-prune-watchdog.service -│ ├── backup/ # Backup automation -│ │ └── vzdump-docker-hook.sh -│ ├── network/ # Network configuration -│ │ ├── mtu_phase1.sh -│ │ ├── mtu_phase2.sh -│ │ └── mtu_rollback.sh -│ └── fixes/ # One-time boot fixes -│ └── rc.local.fix.sh -│ -├── SCRIPTS-README.md # Detailed scripts documentation -├── README.md # This file -└── .gitignore # Don't commit secrets, logs, etc. - +├── traefik.yaml # Reverse proxy & load balancer +├── auth.yaml # Authentik authentication +├── postgresql.yaml # PostgreSQL database +├── maintenance.yaml # Cronicle scheduler, Uptime Kuma +├── ... (other service stacks) +└── README.md # This file ``` --- ## 🚀 Quick Start -### Using Docker Swarm Compose Files +### Deploy a Stack ```bash -# SSH into a Docker LXC -ssh root@docker-1 # or docker-2, docker-3 +# SSH into a Docker LXC (docker-1, docker-2, or docker-3) +ssh root@docker-1 -# Clone the repo locally +# Clone this repo locally cd /volume1/docker git clone https://git.bryanmail.net/admin/compose-files.git repo cd repo @@ -57,230 +36,151 @@ cd repo docker stack deploy -c traefik.yaml traefik docker stack deploy -c auth.yaml auth docker stack deploy -c postgresql.yaml postgresql +``` -# View running stacks -docker stack ls +### Update a Stack + +```bash +# Pull latest changes +git pull origin main + +# Re-deploy (applies changes) +docker stack deploy -c traefik.yaml traefik + +# View status +docker stack ps traefik docker service ls ``` -### Managing & Updating Stacks +### Remove a Stack ```bash -# Pull latest from repo -git pull origin main - -# Update a specific stack (re-deploy) -docker stack deploy -c traefik.yaml traefik - -# Remove a stack docker stack rm traefik - -# View stack status -docker stack ps traefik - -# View service logs -docker service logs traefik_reverse-proxy ``` --- -## 🛠️ Operational Scripts +## 📋 Available Stacks -### **[→ Full Scripts Documentation](SCRIPTS-README.md)** - -Quick reference: - -#### 1. **Docker Prune Watchdog** -Automatically manages disk space on Docker nodes. -```bash -# Check status -systemctl status docker-prune-watchdog.service - -# View logs -tail -f /dev/shm/docker-prune-watchdog.log - -# Configuration -cat /etc/docker-prune-watchdog.env -``` - -#### 2. **Proxmox Backup Hook** -Gracefully stops Docker during backups to prevent corruption. -```bash -# Installed at -/usr/local/bin/vzdump-docker-hook.sh - -# Check logs -tail -f /var/log/vzdump-docker-hook.log -``` - -#### 3. **MTU 9000 Migration** -Migrate Docker Swarm to Jumbo Frames for improved performance. -```bash -# See SCRIPTS-README.md for detailed steps -``` +| Stack | File | Purpose | +|-------|------|---------| +| Traefik | traefik.yaml | Reverse proxy, load balancer, TLS termination | +| Authentik | auth.yaml | Authentication & authorization | +| PostgreSQL | postgresql.yaml | Database backend | +| Maintenance | maintenance.yaml | Cronicle jobs, Uptime Kuma monitoring | +| ... | ... | (Add more as you create them) | --- -## 📋 Architecture - -### Proxmox Cluster -``` -nuck7-1 (Hypervisor) nuck7-2 (Hypervisor) nuck7-3 (Hypervisor) -├─ docker-1 (LXC 4031) ├─ docker-2 (LXC 4032) ├─ docker-3 (LXC 4033) -│ └─ Docker Swarm Node │ └─ Docker Swarm Node │ └─ Docker Swarm Node -│ (Manager) │ (Leader) │ (Manager) -└─ ...other LXCs └─ ...other LXCs └─ ...other LXCs -``` - -### Storage -- **CephFS** mounted at `/volume1/docker/` (shared across all docker LXCs) -- **Compose files**: `/volume1/docker/compose-files/` -- **Scripts**: `/volume1/docker/mtu_migration/`, `/usr/local/bin/`, etc. -- **Data volumes**: PostgreSQL, Authentik, and other services use named volumes or `/volume1/docker/` mounts - -### Networking -- **VIP**: 192.168.4.30 (Keepalived, managed by docker-2) -- **Docker hosts**: 192.168.4.31 (docker-1), 192.168.4.32 (docker-2), 192.168.4.33 (docker-3) -- **Traefik**: Reverse proxy with Let's Encrypt TLS + Authentik forward auth -- **Domain**: bryanmail.net - ---- - -## 🔧 Common Tasks +## 🛠️ Common Tasks ### Deploy a New Service -1. **Create compose file** in `/volume1/docker/compose-files/myservice.yaml` -2. **Test locally**: +1. **Create compose file** in this repo: `myservice.yaml` +2. **Test locally** (on single host): ```bash - docker compose -f myservice.yaml up -d - docker compose -f myservice.yaml logs + docker-compose -f myservice.yaml up -d ``` -3. **Convert to Swarm** (remove `container_name`, use `services:` format for Swarm) +3. **Convert to Swarm format** (remove `container_name`, use `services:` for Swarm) 4. **Deploy to Swarm**: ```bash docker stack deploy -c myservice.yaml myservice ``` -5. **Commit to repo**: +5. **Commit & push**: ```bash git add myservice.yaml git commit -m "Add myservice stack" git push origin main ``` -### Update a Running Service - -```bash -# Edit the compose file -nano /volume1/docker/compose-files/traefik.yaml - -# Redeploy -docker stack deploy -c /volume1/docker/compose-files/traefik.yaml traefik - -# Verify changes -docker stack ps traefik -``` - -### Check Disk Space - -```bash -# On docker host -df -h /volume1/docker-root - -# View prune watchdog status -systemctl status docker-prune-watchdog.service -tail -f /dev/shm/docker-prune-watchdog.log -``` - -### Monitor Services +### Check Service Status ```bash # List all services docker service ls -# Get service details +# Get details about a service docker service inspect traefik_reverse-proxy # View service logs docker service logs -f traefik_reverse-proxy -# Check service tasks (containers) +# Check tasks (containers) docker service ps traefik_reverse-proxy ``` +### Monitor Disk Space + +```bash +df -h /volume1/docker-root + +# Docker prune watchdog handles auto-cleanup (see homelab-scripts repo) +``` + --- -## ⚠️ Important Notes +## 🔐 Secrets Management -### Secrets Management **DO NOT commit secrets, passwords, or API keys to this repo.** Use one of these approaches: -1. **Docker Secrets** (for Swarm) - ```yaml - services: - myapp: - secrets: - - db_password - secrets: - db_password: - external: true - ``` - -2. **Environment files** (add to `.gitignore`) - ```bash - # Create .env (not tracked by git) - echo "DB_PASSWORD=mysecretpassword" > .env - - # Use in compose - env_file: .env - ``` -3. **Docker Secrets CLI** - ```bash - echo "mysecretpassword" | docker secret create db_password - - ``` +### Option 1: Docker Secrets (Recommended for Swarm) +```yaml +services: + myapp: + secrets: + - db_password -### Git Workflow - -```bash -# Before starting work -git pull origin main - -# Create a feature branch for new services -git checkout -b feature/new-service - -# Make changes -# ... edit files ... - -# Commit -git add . -git commit -m "Add new-service stack with documentation" - -# Push -git push origin feature/new-service - -# Create PR for review (if using pull requests) +secrets: + db_password: + external: true ``` -### Backup Strategy -- **Proxmox backups**: Automatic via vzdump-docker-hook.sh -- **Database exports**: Automated via Cronicle jobs -- **Configurations**: Version controlled in this repo (compose files, scripts) -- **Volumes**: Proxmox handles via CephFS snapshots +Create the secret: +```bash +echo "mysecretpassword" | docker secret create db_password - +``` + +### Option 2: Environment Files (Not tracked by git) +```bash +# Create .env (add to .gitignore) +echo "DB_PASSWORD=mysecretpassword" > .env + +# Use in compose +env_file: .env +``` --- -## 📚 Related Documentation +## 📚 Architecture -- **Architecture Decision Records (ADRs)**: See notes - - ADR-006: Jumbo Frames (MTU 9000) - - ADR-009: Backup Strategy - - ADR-011: Observability & Self-Healing - -- **Detailed Scripts**: [→ SCRIPTS-README.md](SCRIPTS-README.md) -- **Proxmox Setup**: Documented in separate notes -- **MCP Servers**: Proxmox MCP configured in Open WebUI +### Swarm Cluster +``` +nuck7-1 (Hypervisor) nuck7-2 (Hypervisor) nuck7-3 (Hypervisor) +├─ docker-1 (LXC 4031) ├─ docker-2 (LXC 4032) ├─ docker-3 (LXC 4033) +│ └─ Swarm Manager │ └─ Swarm Leader │ └─ Swarm Manager +└─ ... └─ ... └─ ... +``` + +### Storage +- **CephFS** mounted at `/volume1/docker/` (shared across all nodes) +- **Compose files**: `/volume1/docker/compose-files/` +- **Service data**: Named volumes or `/volume1/docker/` mounts + +### Networking +- **VIP**: 192.168.4.30 (Keepalived) +- **Docker hosts**: 192.168.4.31-33 +- **Traefik**: Reverse proxy with Let's Encrypt TLS +- **Domain**: bryanmail.net + +--- + +## 🔗 Related Repositories + +- **[homelab-scripts](https://git.bryanmail.net/admin/homelab-scripts)** - Operational scripts (backup hooks, monitoring, network config) +- **Proxmox MCP Setup** - Documented in notes +- **Architecture Decision Records (ADRs)** - Documented in notes --- @@ -296,47 +196,62 @@ docker node ls # Check disk space df -h /volume1/docker-root - -# Check Docker status -systemctl status docker ``` -### Service container keeps restarting +### Service keeps crashing ```bash -# View service logs -docker service logs -f myservice_container +# View logs +docker service logs -f myservice_name -# Inspect container state +# Inspect container docker ps -a | grep myservice ``` ### Network issues ```bash -# Check overlay networks +# List networks docker network ls --filter driver=overlay -# Verify network connectivity +# Test connectivity docker run --rm --network traefik_backend alpine ping traefik_reverse-proxy ``` --- -## 🤝 Contributing +## 📞 Contributing When adding new services: -1. **Use Swarm-compatible YAML** (no `container_name`, proper `services:` format) -2. **Document requirements** in compose file comments -3. **Test on non-production first** (docker-compose up on single host) -4. **Add notes** about persistent volumes, secrets, networking -5. **Update this README** if adding infrastructure changes +1. Use **Swarm-compatible YAML** (no `container_name`) +2. Document requirements in compose file comments +3. Test on non-production first +4. Add notes about volumes, secrets, networking +5. Update this README with stack description --- -## 📞 Support +## 📝 Git Workflow -For detailed script documentation: [→ SCRIPTS-README.md](SCRIPTS-README.md) +```bash +# Before starting work +git pull origin main -For architecture decisions: See ADRs in notes +# Create feature branch for new service +git checkout -b feature/new-service -For troubleshooting: Check logs, verify disk space, check Docker Swarm status +# Make changes and commit +git add . +git commit -m "Add new-service stack" + +# Push +git push origin feature/new-service +``` + +--- + +## Version Control Best Practices + +- **Keep compose files in sync** with deployed state +- **Pin image versions** (avoid `latest` tag) +- **Document breaking changes** in commit messages +- **Use meaningful commit messages** for audit trail