# Homelab Infrastructure Repository

This Gitea repository contains all configuration files and scripts for managing the homelab infrastructure:

- **Docker Swarm** cluster (3 nodes: docker-1, docker-2, docker-3)
- **Proxmox** cluster (3 nodes: nuck7-1, nuck7-2, nuck7-3)
- **Operational automation** (backups, monitoring, network management)

---

## 📁 Directory Structure

```
├── compose-files/              # Docker Swarm compose files
│   ├── traefik.yaml           # Reverse proxy & load balancer
│   ├── auth.yaml              # Authentik authentication
│   ├── postgresql.yaml        # PostgreSQL database
│   ├── ... (other stacks)
│   └── (add new services here)
│
├── scripts/                    # Operational scripts
│   ├── monitoring/            # Disk & service monitoring
│   │   ├── docker-prune-watchdog.sh
│   │   ├── docker-prune-watchdog.env.example
│   │   └── docker-prune-watchdog.service
│   ├── backup/                # Backup automation
│   │   └── vzdump-docker-hook.sh
│   ├── network/               # Network configuration
│   │   ├── mtu_phase1.sh
│   │   ├── mtu_phase2.sh
│   │   └── mtu_rollback.sh
│   └── fixes/                 # One-time boot fixes
│       └── rc.local.fix.sh
│
├── SCRIPTS-README.md          # Detailed scripts documentation
├── README.md                  # This file
└── .gitignore                 # Don't commit secrets, logs, etc.

```

---

## 🚀 Quick Start

### Using Docker Swarm Compose Files

```bash
# SSH into a Docker LXC
ssh root@docker-1  # or docker-2, docker-3

# Clone the repo locally
cd /volume1/docker
git clone https://git.bryanmail.net/admin/compose-files.git repo
cd repo

# Deploy a stack
docker stack deploy -c traefik.yaml traefik
docker stack deploy -c auth.yaml auth
docker stack deploy -c postgresql.yaml postgresql

# View running stacks
docker stack ls
docker service ls
```

### Managing & Updating Stacks

```bash
# Pull latest from repo
git pull origin main

# Update a specific stack (re-deploy)
docker stack deploy -c traefik.yaml traefik

# Remove a stack
docker stack rm traefik

# View stack status
docker stack ps traefik

# View service logs
docker service logs traefik_reverse-proxy
```

---

## 🛠️ Operational Scripts

### **[→ Full Scripts Documentation](SCRIPTS-README.md)**

Quick reference:

#### 1. **Docker Prune Watchdog**
Automatically manages disk space on Docker nodes.
```bash
# Check status
systemctl status docker-prune-watchdog.service

# View logs
tail -f /dev/shm/docker-prune-watchdog.log

# Configuration
cat /etc/docker-prune-watchdog.env
```

#### 2. **Proxmox Backup Hook**
Gracefully stops Docker during backups to prevent corruption.
```bash
# Installed at
/usr/local/bin/vzdump-docker-hook.sh

# Check logs
tail -f /var/log/vzdump-docker-hook.log
```

#### 3. **MTU 9000 Migration**
Migrate Docker Swarm to Jumbo Frames for improved performance.
```bash
# See SCRIPTS-README.md for detailed steps
```

---

## 📋 Architecture

### Proxmox Cluster
```
nuck7-1 (Hypervisor)        nuck7-2 (Hypervisor)        nuck7-3 (Hypervisor)
├─ docker-1 (LXC 4031)      ├─ docker-2 (LXC 4032)      ├─ docker-3 (LXC 4033)
│  └─ Docker Swarm Node     │  └─ Docker Swarm Node     │  └─ Docker Swarm Node
│     (Manager)             │     (Leader)               │     (Manager)
└─ ...other LXCs            └─ ...other LXCs            └─ ...other LXCs
```

### Storage
- **CephFS** mounted at `/volume1/docker/` (shared across all docker LXCs)
- **Compose files**: `/volume1/docker/compose-files/`
- **Scripts**: `/volume1/docker/mtu_migration/`, `/usr/local/bin/`, etc.
- **Data volumes**: PostgreSQL, Authentik, and other services use named volumes or `/volume1/docker/` mounts

### Networking
- **VIP**: 192.168.4.30 (Keepalived, managed by docker-2)
- **Docker hosts**: 192.168.4.31 (docker-1), 192.168.4.32 (docker-2), 192.168.4.33 (docker-3)
- **Traefik**: Reverse proxy with Let's Encrypt TLS + Authentik forward auth
- **Domain**: bryanmail.net

---

## 🔧 Common Tasks

### Deploy a New Service

1. **Create compose file** in `/volume1/docker/compose-files/myservice.yaml`
2. **Test locally**:
   ```bash
   docker compose -f myservice.yaml up -d
   docker compose -f myservice.yaml logs
   ```
3. **Convert to Swarm** (remove `container_name`, use `services:` format for Swarm)
4. **Deploy to Swarm**:
   ```bash
   docker stack deploy -c myservice.yaml myservice
   ```
5. **Commit to repo**:
   ```bash
   git add myservice.yaml
   git commit -m "Add myservice stack"
   git push origin main
   ```

### Update a Running Service

```bash
# Edit the compose file
nano /volume1/docker/compose-files/traefik.yaml

# Redeploy
docker stack deploy -c /volume1/docker/compose-files/traefik.yaml traefik

# Verify changes
docker stack ps traefik
```

### Check Disk Space

```bash
# On docker host
df -h /volume1/docker-root

# View prune watchdog status
systemctl status docker-prune-watchdog.service
tail -f /dev/shm/docker-prune-watchdog.log
```

### Monitor Services

```bash
# List all services
docker service ls

# Get service details
docker service inspect traefik_reverse-proxy

# View service logs
docker service logs -f traefik_reverse-proxy

# Check service tasks (containers)
docker service ps traefik_reverse-proxy
```

---

## ⚠️ Important Notes

### Secrets Management
**DO NOT commit secrets, passwords, or API keys to this repo.**

Use one of these approaches:
1. **Docker Secrets** (for Swarm)
   ```yaml
   services:
     myapp:
       secrets:
         - db_password
   secrets:
     db_password:
       external: true
   ```
   
2. **Environment files** (add to `.gitignore`)
   ```bash
   # Create .env (not tracked by git)
   echo "DB_PASSWORD=mysecretpassword" > .env
   
   # Use in compose
   env_file: .env
   ```

3. **Docker Secrets CLI**
   ```bash
   echo "mysecretpassword" | docker secret create db_password -
   ```

### Git Workflow

```bash
# Before starting work
git pull origin main

# Create a feature branch for new services
git checkout -b feature/new-service

# Make changes
# ... edit files ...

# Commit
git add .
git commit -m "Add new-service stack with documentation"

# Push
git push origin feature/new-service

# Create PR for review (if using pull requests)
```

### Backup Strategy
- **Proxmox backups**: Automatic via vzdump-docker-hook.sh
- **Database exports**: Automated via Cronicle jobs
- **Configurations**: Version controlled in this repo (compose files, scripts)
- **Volumes**: Proxmox handles via CephFS snapshots

---

## 📚 Related Documentation

- **Architecture Decision Records (ADRs)**: See notes
  - ADR-006: Jumbo Frames (MTU 9000)
  - ADR-009: Backup Strategy
  - ADR-011: Observability & Self-Healing
  
- **Detailed Scripts**: [→ SCRIPTS-README.md](SCRIPTS-README.md)
- **Proxmox Setup**: Documented in separate notes
- **MCP Servers**: Proxmox MCP configured in Open WebUI

---

## 🐛 Troubleshooting

### Stack won't deploy
```bash
# Check syntax
docker-compose config -f myservice.yaml

# Check node availability
docker node ls

# Check disk space
df -h /volume1/docker-root

# Check Docker status
systemctl status docker
```

### Service container keeps restarting
```bash
# View service logs
docker service logs -f myservice_container

# Inspect container state
docker ps -a | grep myservice
```

### Network issues
```bash
# Check overlay networks
docker network ls --filter driver=overlay

# Verify network connectivity
docker run --rm --network traefik_backend alpine ping traefik_reverse-proxy
```

---

## 🤝 Contributing

When adding new services:

1. **Use Swarm-compatible YAML** (no `container_name`, proper `services:` format)
2. **Document requirements** in compose file comments
3. **Test on non-production first** (docker-compose up on single host)
4. **Add notes** about persistent volumes, secrets, networking
5. **Update this README** if adding infrastructure changes

---

## 📞 Support

For detailed script documentation: [→ SCRIPTS-README.md](SCRIPTS-README.md)

For architecture decisions: See ADRs in notes

For troubleshooting: Check logs, verify disk space, check Docker Swarm status