# Traefik Log Rotation

## Why this exists

`traefik/traefik.yaml` runs Traefik with:
- `--accesslog.filePath=/traefik/logs/access.log`
- `--log.filePath=/traefik/logs/traefik.log`
- `--log.level=DEBUG`

Neither file has any built-in rotation -- Traefik has no native rotate-on-size
nor a SIGUSR1/reopen handler. Docker's `json-file` log-driver rotation
(`max-size`/`max-file`) only applies to stdout, not to files Traefik writes
directly via `--accesslog.filePath`/`--log.filePath`. Result: `access.log` grew
to **~15.5GB unrotated** before this was caught, on a CephFS volume
(`/volume1/docker-root`) already at 82-84% used. Both the disk pressure and
the ongoing write latency of appending to a 15GB file on a network filesystem
on every request through Traefik were flagged as a real risk factor during
troubleshooting (Uptime Kuma WebSocket flapping investigation, Sep 2026).

**Note:** during that investigation, the actual root cause of the WebSocket
flapping turned out to be Uptime Kuma monitor misconfigurations (a Postgres
monitor throwing a null-reference error, and several monitors failing TLS
validation against self-signed/internal-IP certs) -- not this log file. This
rotation fix is still worth doing as general disk/IO hygiene, just not
causally tied to that incident.

## Architecture constraint this design accounts for

`traefik_reverse-proxy` runs Swarm *`mode: global`* -- one instance on **each**
of docker-1, docker-2, docker-3. All three write to the **same physical file**
via the shared CephFS bind mount `/volume1/docker/traefik -> /traefik`
(identical mount, visible identically from any node). That rules out:

- **Signal-based rotation** (classic `create` + `postrotate` sending SIGUSR1):
  Traefik doesn't implement a reopen signal, and even if it did, you'd need to
  signal 3 separate per-node containers in lockstep.
- **Running logrotate on just one node**: works until that node is down, then
  rotation silently stops with no alert.

So this setup uses:
1. **`copytruncate`** (see `traefik-logs.conf`) -- all three Traefik processes
   keep writing to the same inode, no signaling needed. Tradeoff: a few log
   lines written in the exact copy/truncate instant can be lost -- fine for
   diagnostic logs.
2. **Cron on all three nodes**, coordinated via a shared `flock` + shared
   logrotate state file, both also on the CephFS mount (see
   `traefik-logrotate.sh`). Whichever node's cron fires first grabs the lock,
   rotates if due, and updates the shared state so the other two nodes' cron
   runs see it's already done. No single node is a rotation SPOF.
3. **Size-triggered** (`size 250M`) rather than calendar-triggered (`daily`) --
   this file can grow fast under bursts (see incident background above); a
   purely daily interval would still let it balloon between runs. Cron checks
   every 15 minutes, so it cannot grow much past the 250M threshold in practice.

## Install (one-time, per node)

Must run on **all three** docker LXCs -- this is host-level cron/logrotate
config, not something `stack-deploy.sh` can reach (it only touches Swarm
services, not host cron jobs).

```bash
# On each of docker-1 (192.168.4.31), docker-2 (192.168.4.32), docker-3 (192.168.4.33):
ssh root@<192.168.4.31|.32|.33>
cd /volume1/docker/compose-files
bash deploy/git-guard.sh         # confirm sync first, as always
sudo bash traefik/logrotate/install.sh
```

Verify on each node:
```bash
cat /etc/cron.d/traefik-logrotate
logrotate -d /etc/logrotate.d/traefik-logs   # dry-run, confirms syntax
```

## Bootstrapping -- shrinking the *existing* oversized log file

**Not done automatically by `install.sh`.** The new size-triggered config only
prevents *future* unbounded growth -- it won't touch the current 15.5GB file
until the next time it crosses 250M (i.e. never, since it's already well past
that and logrotate only acts on crossing the threshold going forward from its
recorded size at last check).

`copytruncate` always copies the full current file before truncating it --
that's inherent to how it works, not a bug. Forcing a rotation of the current
15.5GB file would momentarily need roughly another 15-GB-sized chunk of free
space -- and `/volume1/docker-root` only had **~14G free** at last check.
Doing this blindly could tip the volume to 100% mid-operation.

**Recommended manual step (operator-run, on any one node -- it's the same
CephFS file from all three)**: these are diagnostic access/debug logs, not
something worth preserving in full, so just truncate directly rather than
compress-then-truncate:

```bash
# Optional: keep a small tail sample for reference before truncating
tail -c 50000000 /volume1/docker/traefik/logs/access.log > \
  /volume1/docker/traefik-access-log-archive-$(date +%Y%m%d).log

# Then truncate in place (safe -- Traefik's existing file handles on all 3
# nodes stay valid, same as copytruncate's own mechanism):
: > /volume1/docker/traefik/logs/access.log
: > /volume1/docker/traefik/logs/traefik.log

# Confirm:
df -h /volume1/docker-root
ls -la /volume1/docker/traefik/logs/
```

After this one-time bootstrap, the cron+logrotate setup keeps it bounded
(rotates at 250M, keeps 48 compressed generations, prunes anything over 14
days old) going forward without needing any further manual intervention.

## Related, NOT included in this change (follow-up to consider separately)

`traefik.yaml` currently runs `--log.level=DEBUG` -- verbose debug logging in
production, which is a meaningful contributor to how fast these files grow.
Lowering to `INFO` would reduce volume significantly but requires a real
modification + redeploy of the high-blast-radius `traefik` stack (all
HTTP/HTTPS routing depends on it), so it's intentionally left out of this PR
and should be its own reviewed change if wanted.