121 lines
5.7 KiB
Markdown
121 lines
5.7 KiB
Markdown
# Traefik Log Rotation
|
|
|
|
## Why this exists
|
|
|
|
`traefik/traefik.yaml` runs Traefik with:
|
|
- `--accesslog.filePath=/traefik/logs/access.log`
|
|
- `--log.filePath=/traefik/logs/traefik.log`
|
|
- `--log.level=DEBUG`
|
|
|
|
Neither file has any built-in rotation -- Traefik has no native rotate-on-size
|
|
nor a SIGUSR1/reopen handler. Docker's `json-file` log-driver rotation
|
|
(`max-size`/`max-file`) only applies to stdout, not to files Traefik writes
|
|
directly via `--accesslog.filePath`/`--log.filePath`. Result: `access.log` grew
|
|
to **~15.5GB unrotated** before this was caught, on a CephFS volume
|
|
(`/volume1/docker-root`) already at 82-84% used. Both the disk pressure and
|
|
the ongoing write latency of appending to a 15GB file on a network filesystem
|
|
on every request through Traefik were flagged as a real risk factor during
|
|
troubleshooting (Uptime Kuma WebSocket flapping investigation, Sep 2026).
|
|
|
|
**Note:** during that investigation, the actual root cause of the WebSocket
|
|
flapping turned out to be Uptime Kuma monitor misconfigurations (a Postgres
|
|
monitor throwing a null-reference error, and several monitors failing TLS
|
|
validation against self-signed/internal-IP certs) -- not this log file. This
|
|
rotation fix is still worth doing as general disk/IO hygiene, just not
|
|
causally tied to that incident.
|
|
|
|
## Architecture constraint this design accounts for
|
|
|
|
`traefik_reverse-proxy` runs Swarm *`mode: global`* -- one instance on **each**
|
|
of docker-1, docker-2, docker-3. All three write to the **same physical file**
|
|
via the shared CephFS bind mount `/volume1/docker/traefik -> /traefik`
|
|
(identical mount, visible identically from any node). That rules out:
|
|
|
|
- **Signal-based rotation** (classic `create` + `postrotate` sending SIGUSR1):
|
|
Traefik doesn't implement a reopen signal, and even if it did, you'd need to
|
|
signal 3 separate per-node containers in lockstep.
|
|
- **Running logrotate on just one node**: works until that node is down, then
|
|
rotation silently stops with no alert.
|
|
|
|
So this setup uses:
|
|
1. **`copytruncate`** (see `traefik-logs.conf`) -- all three Traefik processes
|
|
keep writing to the same inode, no signaling needed. Tradeoff: a few log
|
|
lines written in the exact copy/truncate instant can be lost -- fine for
|
|
diagnostic logs.
|
|
2. **Cron on all three nodes**, coordinated via a shared `flock` + shared
|
|
logrotate state file, both also on the CephFS mount (see
|
|
`traefik-logrotate.sh`). Whichever node's cron fires first grabs the lock,
|
|
rotates if due, and updates the shared state so the other two nodes' cron
|
|
runs see it's already done. No single node is a rotation SPOF.
|
|
3. **Size-triggered** (`size 250M`) rather than calendar-triggered (`daily`) --
|
|
this file can grow fast under bursts (see incident background above); a
|
|
purely daily interval would still let it balloon between runs. Cron checks
|
|
every 15 minutes, so it cannot grow much past the 250M threshold in practice.
|
|
|
|
## Install (one-time, per node)
|
|
|
|
Must run on **all three** docker LXCs -- this is host-level cron/logrotate
|
|
config, not something `stack-deploy.sh` can reach (it only touches Swarm
|
|
services, not host cron jobs).
|
|
|
|
```bash
|
|
# On each of docker-1 (192.168.4.31), docker-2 (192.168.4.32), docker-3 (192.168.4.33):
|
|
ssh root@<192.168.4.31|.32|.33>
|
|
cd /volume1/docker/compose-files
|
|
bash deploy/git-guard.sh # confirm sync first, as always
|
|
sudo bash traefik/logrotate/install.sh
|
|
```
|
|
|
|
Verify on each node:
|
|
```bash
|
|
cat /etc/cron.d/traefik-logrotate
|
|
logrotate -d /etc/logrotate.d/traefik-logs # dry-run, confirms syntax
|
|
```
|
|
|
|
## Bootstrapping -- shrinking the *existing* oversized log file
|
|
|
|
**Not done automatically by `install.sh`.** The new size-triggered config only
|
|
prevents *future* unbounded growth -- it won't touch the current 15.5GB file
|
|
until the next time it crosses 250M (i.e. never, since it's already well past
|
|
that and logrotate only acts on crossing the threshold going forward from its
|
|
recorded size at last check).
|
|
|
|
`copytruncate` always copies the full current file before truncating it --
|
|
that's inherent to how it works, not a bug. Forcing a rotation of the current
|
|
15.5GB file would momentarily need roughly another 15-GB-sized chunk of free
|
|
space -- and `/volume1/docker-root` only had **~14G free** at last check.
|
|
Doing this blindly could tip the volume to 100% mid-operation.
|
|
|
|
**Recommended manual step (operator-run, on any one node -- it's the same
|
|
CephFS file from all three)**: these are diagnostic access/debug logs, not
|
|
something worth preserving in full, so just truncate directly rather than
|
|
compress-then-truncate:
|
|
|
|
```bash
|
|
# Optional: keep a small tail sample for reference before truncating
|
|
tail -c 50000000 /volume1/docker/traefik/logs/access.log > \
|
|
/volume1/docker/traefik-access-log-archive-$(date +%Y%m%d).log
|
|
|
|
# Then truncate in place (safe -- Traefik's existing file handles on all 3
|
|
# nodes stay valid, same as copytruncate's own mechanism):
|
|
: > /volume1/docker/traefik/logs/access.log
|
|
: > /volume1/docker/traefik/logs/traefik.log
|
|
|
|
# Confirm:
|
|
df -h /volume1/docker-root
|
|
ls -la /volume1/docker/traefik/logs/
|
|
```
|
|
|
|
After this one-time bootstrap, the cron+logrotate setup keeps it bounded
|
|
(rotates at 250M, keeps 48 compressed generations, prunes anything over 14
|
|
days old) going forward without needing any further manual intervention.
|
|
|
|
## Related, NOT included in this change (follow-up to consider separately)
|
|
|
|
`traefik.yaml` currently runs `--log.level=DEBUG` -- verbose debug logging in
|
|
production, which is a meaningful contributor to how fast these files grow.
|
|
Lowering to `INFO` would reduce volume significantly but requires a real
|
|
modification + redeploy of the high-blast-radius `traefik` stack (all
|
|
HTTP/HTTPS routing depends on it), so it's intentionally left out of this PR
|
|
and should be its own reviewed change if wanted.
|