compose-files/traefik/logrotate/README.md

Traefik Log Rotation

Why this exists

traefik/traefik.yaml runs Traefik with:

• --accesslog.filePath=/traefik/logs/access.log
• --log.filePath=/traefik/logs/traefik.log
• --log.level=DEBUG

Neither file has any built-in rotation -- Traefik has no native rotate-on-size nor a SIGUSR1/reopen handler. Docker's json-file log-driver rotation (max-size/max-file) only applies to stdout, not to files Traefik writes directly via --accesslog.filePath/--log.filePath. Result: access.log grew to ~15.5GB unrotated before this was caught, on a CephFS volume (/volume1/docker-root) already at 82-84% used. Both the disk pressure and the ongoing write latency of appending to a 15GB file on a network filesystem on every request through Traefik were flagged as a real risk factor during troubleshooting (Uptime Kuma WebSocket flapping investigation, Sep 2026).

Note: during that investigation, the actual root cause of the WebSocket flapping turned out to be Uptime Kuma monitor misconfigurations (a Postgres monitor throwing a null-reference error, and several monitors failing TLS validation against self-signed/internal-IP certs) -- not this log file. This rotation fix is still worth doing as general disk/IO hygiene, just not causally tied to that incident.

Architecture constraint this design accounts for

traefik_reverse-proxy runs Swarm mode: global -- one instance on each of docker-1, docker-2, docker-3. All three write to the same physical file via the shared CephFS bind mount /volume1/docker/traefik -> /traefik (identical mount, visible identically from any node). That rules out:

• Signal-based rotation (classic create + postrotate sending SIGUSR1): Traefik doesn't implement a reopen signal, and even if it did, you'd need to signal 3 separate per-node containers in lockstep.
• Running logrotate on just one node: works until that node is down, then rotation silently stops with no alert.

So this setup uses:

1. copytruncate (see traefik-logs.conf) -- all three Traefik processes keep writing to the same inode, no signaling needed. Tradeoff: a few log lines written in the exact copy/truncate instant can be lost -- fine for diagnostic logs.
2. Cron on all three nodes, coordinated via a shared flock + shared logrotate state file, both also on the CephFS mount (see traefik-logrotate.sh). Whichever node's cron fires first grabs the lock, rotates if due, and updates the shared state so the other two nodes' cron runs see it's already done. No single node is a rotation SPOF.
3. Size-triggered (size 250M) rather than calendar-triggered (daily) -- this file can grow fast under bursts (see incident background above); a purely daily interval would still let it balloon between runs. Cron checks every 15 minutes, so it cannot grow much past the 250M threshold in practice.

Install (one-time, per node)

Must run on all three docker LXCs -- this is host-level cron/logrotate config, not something stack-deploy.sh can reach (it only touches Swarm services, not host cron jobs).

# On each of docker-1 (192.168.4.31), docker-2 (192.168.4.32), docker-3 (192.168.4.33):
ssh root@<192.168.4.31|.32|.33>
cd /volume1/docker/compose-files
bash deploy/git-guard.sh         # confirm sync first, as always
sudo bash traefik/logrotate/install.sh

Verify on each node:

cat /etc/cron.d/traefik-logrotate
logrotate -d /etc/logrotate.d/traefik-logs   # dry-run, confirms syntax

Bootstrapping -- shrinking the existing oversized log file

Not done automatically by install.sh. The new size-triggered config only prevents future unbounded growth -- it won't touch the current 15.5GB file until the next time it crosses 250M (i.e. never, since it's already well past that and logrotate only acts on crossing the threshold going forward from its recorded size at last check).

copytruncate always copies the full current file before truncating it -- that's inherent to how it works, not a bug. Forcing a rotation of the current 15.5GB file would momentarily need roughly another 15-GB-sized chunk of free space -- and /volume1/docker-root only had ~14G free at last check. Doing this blindly could tip the volume to 100% mid-operation.

Recommended manual step (operator-run, on any one node -- it's the same CephFS file from all three): these are diagnostic access/debug logs, not something worth preserving in full, so just truncate directly rather than compress-then-truncate:

# Optional: keep a small tail sample for reference before truncating
tail -c 50000000 /volume1/docker/traefik/logs/access.log > \
  /volume1/docker/traefik-access-log-archive-$(date +%Y%m%d).log

# Then truncate in place (safe -- Traefik's existing file handles on all 3
# nodes stay valid, same as copytruncate's own mechanism):
: > /volume1/docker/traefik/logs/access.log
: > /volume1/docker/traefik/logs/traefik.log

# Confirm:
df -h /volume1/docker-root
ls -la /volume1/docker/traefik/logs/

After this one-time bootstrap, the cron+logrotate setup keeps it bounded (rotates at 250M, keeps 48 compressed generations, prunes anything over 14 days old) going forward without needing any further manual intervention.

Related, NOT included in this change (follow-up to consider separately)

traefik.yaml currently runs --log.level=DEBUG -- verbose debug logging in production, which is a meaningful contributor to how fast these files grow. Lowering to INFO would reduce volume significantly but requires a real modification + redeploy of the high-blast-radius traefik stack (all HTTP/HTTPS routing depends on it), so it's intentionally left out of this PR and should be its own reviewed change if wanted.