Homelab Infrastructure Repository

This Gitea repository contains all configuration files and scripts for managing the homelab infrastructure:

• Docker Swarm cluster (3 nodes: docker-1, docker-2, docker-3)
• Proxmox cluster (3 nodes: nuck7-1, nuck7-2, nuck7-3)
• Operational automation (backups, monitoring, network management)

---

📁 Directory Structure

├── compose-files/              # Docker Swarm compose files
│   ├── traefik.yaml           # Reverse proxy & load balancer
│   ├── auth.yaml              # Authentik authentication
│   ├── postgresql.yaml        # PostgreSQL database
│   ├── ... (other stacks)
│   └── (add new services here)
│
├── scripts/                    # Operational scripts
│   ├── monitoring/            # Disk & service monitoring
│   │   ├── docker-prune-watchdog.sh
│   │   ├── docker-prune-watchdog.env.example
│   │   └── docker-prune-watchdog.service
│   ├── backup/                # Backup automation
│   │   └── vzdump-docker-hook.sh
│   ├── network/               # Network configuration
│   │   ├── mtu_phase1.sh
│   │   ├── mtu_phase2.sh
│   │   └── mtu_rollback.sh
│   └── fixes/                 # One-time boot fixes
│       └── rc.local.fix.sh
│
├── SCRIPTS-README.md          # Detailed scripts documentation
├── README.md                  # This file
└── .gitignore                 # Don't commit secrets, logs, etc.

---

🚀 Quick Start

Using Docker Swarm Compose Files

# SSH into a Docker LXC
ssh root@docker-1  # or docker-2, docker-3

# Clone the repo locally
cd /volume1/docker
git clone https://git.bryanmail.net/admin/compose-files.git repo
cd repo

# Deploy a stack
docker stack deploy -c traefik.yaml traefik
docker stack deploy -c auth.yaml auth
docker stack deploy -c postgresql.yaml postgresql

# View running stacks
docker stack ls
docker service ls

Managing & Updating Stacks

# Pull latest from repo
git pull origin main

# Update a specific stack (re-deploy)
docker stack deploy -c traefik.yaml traefik

# Remove a stack
docker stack rm traefik

# View stack status
docker stack ps traefik

# View service logs
docker service logs traefik_reverse-proxy

---

🛠️ Operational Scripts

→ Full Scripts Documentation

Quick reference:

1. Docker Prune Watchdog

Automatically manages disk space on Docker nodes.

# Check status
systemctl status docker-prune-watchdog.service

# View logs
tail -f /dev/shm/docker-prune-watchdog.log

# Configuration
cat /etc/docker-prune-watchdog.env

2. Proxmox Backup Hook

Gracefully stops Docker during backups to prevent corruption.

# Installed at
/usr/local/bin/vzdump-docker-hook.sh

# Check logs
tail -f /var/log/vzdump-docker-hook.log

3. MTU 9000 Migration

Migrate Docker Swarm to Jumbo Frames for improved performance.

# See SCRIPTS-README.md for detailed steps

---

📋 Architecture

Proxmox Cluster

nuck7-1 (Hypervisor)        nuck7-2 (Hypervisor)        nuck7-3 (Hypervisor)
├─ docker-1 (LXC 4031)      ├─ docker-2 (LXC 4032)      ├─ docker-3 (LXC 4033)
│  └─ Docker Swarm Node     │  └─ Docker Swarm Node     │  └─ Docker Swarm Node
│     (Manager)             │     (Leader)               │     (Manager)
└─ ...other LXCs            └─ ...other LXCs            └─ ...other LXCs

Storage

• CephFS mounted at /volume1/docker/ (shared across all docker LXCs)
• Compose files: /volume1/docker/compose-files/
• Scripts: /volume1/docker/mtu_migration/, /usr/local/bin/, etc.
• Data volumes: PostgreSQL, Authentik, and other services use named volumes or /volume1/docker/ mounts

Networking

• VIP: 192.168.4.30 (Keepalived, managed by docker-2)
• Docker hosts: 192.168.4.31 (docker-1), 192.168.4.32 (docker-2), 192.168.4.33 (docker-3)
• Traefik: Reverse proxy with Let's Encrypt TLS + Authentik forward auth
• Domain: bryanmail.net

---

🔧 Common Tasks

Deploy a New Service

1. Create compose file in /volume1/docker/compose-files/myservice.yaml
2. Test locally:

   docker compose -f myservice.yaml up -d
   docker compose -f myservice.yaml logs
   

3. Convert to Swarm (remove container_name, use services: format for Swarm)
4. Deploy to Swarm:

   docker stack deploy -c myservice.yaml myservice
   

5. Commit to repo:

   git add myservice.yaml
   git commit -m "Add myservice stack"
   git push origin main
   

Update a Running Service

# Edit the compose file
nano /volume1/docker/compose-files/traefik.yaml

# Redeploy
docker stack deploy -c /volume1/docker/compose-files/traefik.yaml traefik

# Verify changes
docker stack ps traefik

Check Disk Space

# On docker host
df -h /volume1/docker-root

# View prune watchdog status
systemctl status docker-prune-watchdog.service
tail -f /dev/shm/docker-prune-watchdog.log

Monitor Services

# List all services
docker service ls

# Get service details
docker service inspect traefik_reverse-proxy

# View service logs
docker service logs -f traefik_reverse-proxy

# Check service tasks (containers)
docker service ps traefik_reverse-proxy

---

⚠️ Important Notes

Secrets Management

DO NOT commit secrets, passwords, or API keys to this repo.

Use one of these approaches:

1. Docker Secrets (for Swarm)

   services:
     myapp:
       secrets:
         - db_password
   secrets:
     db_password:
       external: true
   

2. Environment files (add to .gitignore)

   # Create .env (not tracked by git)
   echo "DB_PASSWORD=mysecretpassword" > .env
   
   # Use in compose
   env_file: .env
   

3. Docker Secrets CLI

   echo "mysecretpassword" | docker secret create db_password -
   

Git Workflow

# Before starting work
git pull origin main

# Create a feature branch for new services
git checkout -b feature/new-service

# Make changes
# ... edit files ...

# Commit
git add .
git commit -m "Add new-service stack with documentation"

# Push
git push origin feature/new-service

# Create PR for review (if using pull requests)

Backup Strategy

• Proxmox backups: Automatic via vzdump-docker-hook.sh
• Database exports: Automated via Cronicle jobs
• Configurations: Version controlled in this repo (compose files, scripts)
• Volumes: Proxmox handles via CephFS snapshots

---

📚 Related Documentation

• Architecture Decision Records (ADRs): See notes

  • ADR-006: Jumbo Frames (MTU 9000)
  • ADR-009: Backup Strategy
  • ADR-011: Observability & Self-Healing

• Detailed Scripts: → SCRIPTS-README.md

• Proxmox Setup: Documented in separate notes

• MCP Servers: Proxmox MCP configured in Open WebUI

---

🐛 Troubleshooting

Stack won't deploy

# Check syntax
docker-compose config -f myservice.yaml

# Check node availability
docker node ls

# Check disk space
df -h /volume1/docker-root

# Check Docker status
systemctl status docker

Service container keeps restarting

# View service logs
docker service logs -f myservice_container

# Inspect container state
docker ps -a | grep myservice

Network issues

# Check overlay networks
docker network ls --filter driver=overlay

# Verify network connectivity
docker run --rm --network traefik_backend alpine ping traefik_reverse-proxy

---

🤝 Contributing

When adding new services:

1. Use Swarm-compatible YAML (no container_name, proper services: format)
2. Document requirements in compose file comments
3. Test on non-production first (docker-compose up on single host)
4. Add notes about persistent volumes, secrets, networking
5. Update this README if adding infrastructure changes

---

📞 Support

For detailed script documentation: → SCRIPTS-README.md

For architecture decisions: See ADRs in notes

For troubleshooting: Check logs, verify disk space, check Docker Swarm status