Archived
343 lines
8.2 KiB
Markdown
343 lines
8.2 KiB
Markdown
# Homelab Infrastructure Repository
|
|
|
|
This Gitea repository contains all configuration files and scripts for managing the homelab infrastructure:
|
|
|
|
- **Docker Swarm** cluster (3 nodes: docker-1, docker-2, docker-3)
|
|
- **Proxmox** cluster (3 nodes: nuck7-1, nuck7-2, nuck7-3)
|
|
- **Operational automation** (backups, monitoring, network management)
|
|
|
|
---
|
|
|
|
## 📁 Directory Structure
|
|
|
|
```
|
|
├── compose-files/ # Docker Swarm compose files
|
|
│ ├── traefik.yaml # Reverse proxy & load balancer
|
|
│ ├── auth.yaml # Authentik authentication
|
|
│ ├── postgresql.yaml # PostgreSQL database
|
|
│ ├── ... (other stacks)
|
|
│ └── (add new services here)
|
|
│
|
|
├── scripts/ # Operational scripts
|
|
│ ├── monitoring/ # Disk & service monitoring
|
|
│ │ ├── docker-prune-watchdog.sh
|
|
│ │ ├── docker-prune-watchdog.env.example
|
|
│ │ └── docker-prune-watchdog.service
|
|
│ ├── backup/ # Backup automation
|
|
│ │ └── vzdump-docker-hook.sh
|
|
│ ├── network/ # Network configuration
|
|
│ │ ├── mtu_phase1.sh
|
|
│ │ ├── mtu_phase2.sh
|
|
│ │ └── mtu_rollback.sh
|
|
│ └── fixes/ # One-time boot fixes
|
|
│ └── rc.local.fix.sh
|
|
│
|
|
├── SCRIPTS-README.md # Detailed scripts documentation
|
|
├── README.md # This file
|
|
└── .gitignore # Don't commit secrets, logs, etc.
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Using Docker Swarm Compose Files
|
|
|
|
```bash
|
|
# SSH into a Docker LXC
|
|
ssh root@docker-1 # or docker-2, docker-3
|
|
|
|
# Clone the repo locally
|
|
cd /volume1/docker
|
|
git clone https://git.bryanmail.net/admin/compose-files.git repo
|
|
cd repo
|
|
|
|
# Deploy a stack
|
|
docker stack deploy -c traefik.yaml traefik
|
|
docker stack deploy -c auth.yaml auth
|
|
docker stack deploy -c postgresql.yaml postgresql
|
|
|
|
# View running stacks
|
|
docker stack ls
|
|
docker service ls
|
|
```
|
|
|
|
### Managing & Updating Stacks
|
|
|
|
```bash
|
|
# Pull latest from repo
|
|
git pull origin main
|
|
|
|
# Update a specific stack (re-deploy)
|
|
docker stack deploy -c traefik.yaml traefik
|
|
|
|
# Remove a stack
|
|
docker stack rm traefik
|
|
|
|
# View stack status
|
|
docker stack ps traefik
|
|
|
|
# View service logs
|
|
docker service logs traefik_reverse-proxy
|
|
```
|
|
|
|
---
|
|
|
|
## 🛠️ Operational Scripts
|
|
|
|
### **[→ Full Scripts Documentation](SCRIPTS-README.md)**
|
|
|
|
Quick reference:
|
|
|
|
#### 1. **Docker Prune Watchdog**
|
|
Automatically manages disk space on Docker nodes.
|
|
```bash
|
|
# Check status
|
|
systemctl status docker-prune-watchdog.service
|
|
|
|
# View logs
|
|
tail -f /dev/shm/docker-prune-watchdog.log
|
|
|
|
# Configuration
|
|
cat /etc/docker-prune-watchdog.env
|
|
```
|
|
|
|
#### 2. **Proxmox Backup Hook**
|
|
Gracefully stops Docker during backups to prevent corruption.
|
|
```bash
|
|
# Installed at
|
|
/usr/local/bin/vzdump-docker-hook.sh
|
|
|
|
# Check logs
|
|
tail -f /var/log/vzdump-docker-hook.log
|
|
```
|
|
|
|
#### 3. **MTU 9000 Migration**
|
|
Migrate Docker Swarm to Jumbo Frames for improved performance.
|
|
```bash
|
|
# See SCRIPTS-README.md for detailed steps
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 Architecture
|
|
|
|
### Proxmox Cluster
|
|
```
|
|
nuck7-1 (Hypervisor) nuck7-2 (Hypervisor) nuck7-3 (Hypervisor)
|
|
├─ docker-1 (LXC 4031) ├─ docker-2 (LXC 4032) ├─ docker-3 (LXC 4033)
|
|
│ └─ Docker Swarm Node │ └─ Docker Swarm Node │ └─ Docker Swarm Node
|
|
│ (Manager) │ (Leader) │ (Manager)
|
|
└─ ...other LXCs └─ ...other LXCs └─ ...other LXCs
|
|
```
|
|
|
|
### Storage
|
|
- **CephFS** mounted at `/volume1/docker/` (shared across all docker LXCs)
|
|
- **Compose files**: `/volume1/docker/compose-files/`
|
|
- **Scripts**: `/volume1/docker/mtu_migration/`, `/usr/local/bin/`, etc.
|
|
- **Data volumes**: PostgreSQL, Authentik, and other services use named volumes or `/volume1/docker/` mounts
|
|
|
|
### Networking
|
|
- **VIP**: 192.168.4.30 (Keepalived, managed by docker-2)
|
|
- **Docker hosts**: 192.168.4.31 (docker-1), 192.168.4.32 (docker-2), 192.168.4.33 (docker-3)
|
|
- **Traefik**: Reverse proxy with Let's Encrypt TLS + Authentik forward auth
|
|
- **Domain**: bryanmail.net
|
|
|
|
---
|
|
|
|
## 🔧 Common Tasks
|
|
|
|
### Deploy a New Service
|
|
|
|
1. **Create compose file** in `/volume1/docker/compose-files/myservice.yaml`
|
|
2. **Test locally**:
|
|
```bash
|
|
docker compose -f myservice.yaml up -d
|
|
docker compose -f myservice.yaml logs
|
|
```
|
|
3. **Convert to Swarm** (remove `container_name`, use `services:` format for Swarm)
|
|
4. **Deploy to Swarm**:
|
|
```bash
|
|
docker stack deploy -c myservice.yaml myservice
|
|
```
|
|
5. **Commit to repo**:
|
|
```bash
|
|
git add myservice.yaml
|
|
git commit -m "Add myservice stack"
|
|
git push origin main
|
|
```
|
|
|
|
### Update a Running Service
|
|
|
|
```bash
|
|
# Edit the compose file
|
|
nano /volume1/docker/compose-files/traefik.yaml
|
|
|
|
# Redeploy
|
|
docker stack deploy -c /volume1/docker/compose-files/traefik.yaml traefik
|
|
|
|
# Verify changes
|
|
docker stack ps traefik
|
|
```
|
|
|
|
### Check Disk Space
|
|
|
|
```bash
|
|
# On docker host
|
|
df -h /volume1/docker-root
|
|
|
|
# View prune watchdog status
|
|
systemctl status docker-prune-watchdog.service
|
|
tail -f /dev/shm/docker-prune-watchdog.log
|
|
```
|
|
|
|
### Monitor Services
|
|
|
|
```bash
|
|
# List all services
|
|
docker service ls
|
|
|
|
# Get service details
|
|
docker service inspect traefik_reverse-proxy
|
|
|
|
# View service logs
|
|
docker service logs -f traefik_reverse-proxy
|
|
|
|
# Check service tasks (containers)
|
|
docker service ps traefik_reverse-proxy
|
|
```
|
|
|
|
---
|
|
|
|
## ⚠️ Important Notes
|
|
|
|
### Secrets Management
|
|
**DO NOT commit secrets, passwords, or API keys to this repo.**
|
|
|
|
Use one of these approaches:
|
|
1. **Docker Secrets** (for Swarm)
|
|
```yaml
|
|
services:
|
|
myapp:
|
|
secrets:
|
|
- db_password
|
|
secrets:
|
|
db_password:
|
|
external: true
|
|
```
|
|
|
|
2. **Environment files** (add to `.gitignore`)
|
|
```bash
|
|
# Create .env (not tracked by git)
|
|
echo "DB_PASSWORD=mysecretpassword" > .env
|
|
|
|
# Use in compose
|
|
env_file: .env
|
|
```
|
|
|
|
3. **Docker Secrets CLI**
|
|
```bash
|
|
echo "mysecretpassword" | docker secret create db_password -
|
|
```
|
|
|
|
### Git Workflow
|
|
|
|
```bash
|
|
# Before starting work
|
|
git pull origin main
|
|
|
|
# Create a feature branch for new services
|
|
git checkout -b feature/new-service
|
|
|
|
# Make changes
|
|
# ... edit files ...
|
|
|
|
# Commit
|
|
git add .
|
|
git commit -m "Add new-service stack with documentation"
|
|
|
|
# Push
|
|
git push origin feature/new-service
|
|
|
|
# Create PR for review (if using pull requests)
|
|
```
|
|
|
|
### Backup Strategy
|
|
- **Proxmox backups**: Automatic via vzdump-docker-hook.sh
|
|
- **Database exports**: Automated via Cronicle jobs
|
|
- **Configurations**: Version controlled in this repo (compose files, scripts)
|
|
- **Volumes**: Proxmox handles via CephFS snapshots
|
|
|
|
---
|
|
|
|
## 📚 Related Documentation
|
|
|
|
- **Architecture Decision Records (ADRs)**: See notes
|
|
- ADR-006: Jumbo Frames (MTU 9000)
|
|
- ADR-009: Backup Strategy
|
|
- ADR-011: Observability & Self-Healing
|
|
|
|
- **Detailed Scripts**: [→ SCRIPTS-README.md](SCRIPTS-README.md)
|
|
- **Proxmox Setup**: Documented in separate notes
|
|
- **MCP Servers**: Proxmox MCP configured in Open WebUI
|
|
|
|
---
|
|
|
|
## 🐛 Troubleshooting
|
|
|
|
### Stack won't deploy
|
|
```bash
|
|
# Check syntax
|
|
docker-compose config -f myservice.yaml
|
|
|
|
# Check node availability
|
|
docker node ls
|
|
|
|
# Check disk space
|
|
df -h /volume1/docker-root
|
|
|
|
# Check Docker status
|
|
systemctl status docker
|
|
```
|
|
|
|
### Service container keeps restarting
|
|
```bash
|
|
# View service logs
|
|
docker service logs -f myservice_container
|
|
|
|
# Inspect container state
|
|
docker ps -a | grep myservice
|
|
```
|
|
|
|
### Network issues
|
|
```bash
|
|
# Check overlay networks
|
|
docker network ls --filter driver=overlay
|
|
|
|
# Verify network connectivity
|
|
docker run --rm --network traefik_backend alpine ping traefik_reverse-proxy
|
|
```
|
|
|
|
---
|
|
|
|
## 🤝 Contributing
|
|
|
|
When adding new services:
|
|
|
|
1. **Use Swarm-compatible YAML** (no `container_name`, proper `services:` format)
|
|
2. **Document requirements** in compose file comments
|
|
3. **Test on non-production first** (docker-compose up on single host)
|
|
4. **Add notes** about persistent volumes, secrets, networking
|
|
5. **Update this README** if adding infrastructure changes
|
|
|
|
---
|
|
|
|
## 📞 Support
|
|
|
|
For detailed script documentation: [→ SCRIPTS-README.md](SCRIPTS-README.md)
|
|
|
|
For architecture decisions: See ADRs in notes
|
|
|
|
For troubleshooting: Check logs, verify disk space, check Docker Swarm status
|