traefik/traefik.yaml runs Traefik with --accesslog.filePath and --log.filePath pointing at /traefik/logs/{access,traefik}.log
(--log.level=DEBUG). Neither file has any rotation — Traefik has no
native rotate-on-size or reopen-on-signal support, and Docker's json-file log-driver max-size/max-file only caps stdout, not files
written directly via these flags.
Found access.log at ~15.5GB, unrotated, on /volume1/docker-root
(CephFS, 82-84% used) while investigating an unrelated Uptime Kuma
WebSocket-flapping incident. Correction/update: that incident's actual
root cause turned out to be Uptime Kuma monitor misconfigurations (a
Postgres monitor with a null-reference bug, plus several monitors failing
TLS validation against self-signed/internal-IP certs) — confirmed resolved
by the operator independently of this change. This log rotation fix is not causally tied to that incident; it's still worth merging as
standalone disk/IO hygiene given the file size and shared-storage pressure.
Why this needed its own design (not just "add a logrotate conf")
traefik_reverse-proxy runs Swarm mode: global — one Traefik
instance on each of docker-1/2/3, all three writing to the same
physical file via the shared CephFS mount /volume1/docker/traefik -> /traefik. That rules out standard signal-based rotation (no reopen
handler in Traefik, and no clean way to signal 3 independent per-node
containers in lockstep) and rules out running logrotate on just one node
(silent SPOF if that node is down).
Solution:
copytruncate — all three Traefik processes keep appending to the
same inode, no signaling required.
Cron on all three nodes, coordinated via a shared flock + shared
logrotate state file (both on the same CephFS mount) — whichever node's
cron fires first does the rotation; no single node is a SPOF.
Size-triggered (250M), not calendar-triggered — this file can grow
fast under bursts, so a daily rule alone wouldn't bound it well.
traefik/logrotate/README.md — full rationale, install steps, and manual bootstrap instructions for shrinking the existing 15.5GB file (deliberately not automated — needs disk-headroom awareness given /volume1/docker-root is at 82%+)
What's NOT included (by design)
No change to traefik.yaml / no stack redeploy — this is host-level
cron/logrotate config that stack-deploy.sh has no reach into (it only
touches Swarm services). Zero risk to the running traefik stack.
No change to --log.level=DEBUG — flagged in the README as related
follow-up debt, but lowering it means a real modification + redeploy of
a high-blast-radius stack, so it's out of scope here.
No automatic shrink of the existing 15.5GB file — see manual bootstrap
steps in the README; needs a disk-space check first (~14G free at last
check, copytruncate needs roughly the source file's size in headroom).
Rollout (manual, once per node, not via stack-deploy.sh)
ssh root@<192.168.4.31|.32|.33>
cd /volume1/docker/compose-files
bash deploy/git-guard.sh
sudo bash traefik/logrotate/install.sh
Risk
Low — no Swarm service touched, no traefik.yaml change, additive-only
host config. Worst case if misconfigured: rotation doesn't happen (status
quo), not a routing outage.
## Problem
`traefik/traefik.yaml` runs Traefik with `--accesslog.filePath` and
`--log.filePath` pointing at `/traefik/logs/{access,traefik}.log`
(`--log.level=DEBUG`). Neither file has any rotation — Traefik has no
native rotate-on-size or reopen-on-signal support, and Docker's
`json-file` log-driver `max-size`/`max-file` only caps stdout, not files
written directly via these flags.
Found `access.log` at **~15.5GB, unrotated**, on `/volume1/docker-root`
(CephFS, 82-84% used) while investigating an unrelated Uptime Kuma
WebSocket-flapping incident. **Correction/update:** that incident's actual
root cause turned out to be Uptime Kuma monitor misconfigurations (a
Postgres monitor with a null-reference bug, plus several monitors failing
TLS validation against self-signed/internal-IP certs) — confirmed resolved
by the operator independently of this change. This log rotation fix is
*not* causally tied to that incident; it's still worth merging as
standalone disk/IO hygiene given the file size and shared-storage pressure.
## Why this needed its own design (not just "add a logrotate conf")
`traefik_reverse-proxy` runs Swarm **`mode: global`** — one Traefik
instance on **each** of docker-1/2/3, all three writing to the **same
physical file** via the shared CephFS mount `/volume1/docker/traefik ->
/traefik`. That rules out standard signal-based rotation (no reopen
handler in Traefik, and no clean way to signal 3 independent per-node
containers in lockstep) and rules out running logrotate on just one node
(silent SPOF if that node is down).
Solution:
1. **`copytruncate`** — all three Traefik processes keep appending to the
same inode, no signaling required.
2. **Cron on all three nodes**, coordinated via a shared `flock` + shared
logrotate state file (both on the same CephFS mount) — whichever node's
cron fires first does the rotation; no single node is a SPOF.
3. **Size-triggered (250M)**, not calendar-triggered — this file can grow
fast under bursts, so a `daily` rule alone wouldn't bound it well.
Full rationale in `traefik/logrotate/README.md`.
## What's included
- `traefik/logrotate/traefik-logs.conf` — logrotate config
- `traefik/logrotate/traefik-logrotate.sh` — cron-invoked wrapper (flock + shared state)
- `traefik/logrotate/install.sh` — one-time per-node installer
- `traefik/logrotate/README.md` — full rationale, install steps, and manual bootstrap instructions for shrinking the *existing* 15.5GB file (deliberately **not** automated — needs disk-headroom awareness given `/volume1/docker-root` is at 82%+)
## What's NOT included (by design)
- No change to `traefik.yaml` / no stack redeploy — this is host-level
cron/logrotate config that `stack-deploy.sh` has no reach into (it only
touches Swarm services). Zero risk to the running `traefik` stack.
- No change to `--log.level=DEBUG` — flagged in the README as related
follow-up debt, but lowering it means a real modification + redeploy of
a high-blast-radius stack, so it's out of scope here.
- No automatic shrink of the existing 15.5GB file — see manual bootstrap
steps in the README; needs a disk-space check first (~14G free at last
check, `copytruncate` needs roughly the source file's size in headroom).
## Rollout (manual, once per node, not via stack-deploy.sh)
```bash
ssh root@<192.168.4.31|.32|.33>
cd /volume1/docker/compose-files
bash deploy/git-guard.sh
sudo bash traefik/logrotate/install.sh
```
## Risk
Low — no Swarm service touched, no `traefik.yaml` change, additive-only
host config. Worst case if misconfigured: rotation doesn't happen (status
quo), not a routing outage.
Traefik has no built-in log rotation and no SIGUSR1/reopen handling.
--accesslog.filePath and --log.filePath (traefik.yaml) write directly
to /volume1/docker/traefik/logs, which had grown to 15.5GB unrotated,
contributing to disk pressure (docker-2 CephFS at 82%) and adding
write latency to every request through Traefik.
reverse-proxy runs Swarm mode:global (one instance per node), all
writing to the same CephFS file. This wrapper uses a shared flock +
shared logrotate state file (both also on the CephFS mount) so cron
on docker-1/2/3 can run independently without racing or double-rotating.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
traefik/traefik.yamlruns Traefik with--accesslog.filePathand--log.filePathpointing at/traefik/logs/{access,traefik}.log(
--log.level=DEBUG). Neither file has any rotation — Traefik has nonative rotate-on-size or reopen-on-signal support, and Docker's
json-filelog-drivermax-size/max-fileonly caps stdout, not fileswritten directly via these flags.
Found
access.logat ~15.5GB, unrotated, on/volume1/docker-root(CephFS, 82-84% used) while investigating an unrelated Uptime Kuma
WebSocket-flapping incident. Correction/update: that incident's actual
root cause turned out to be Uptime Kuma monitor misconfigurations (a
Postgres monitor with a null-reference bug, plus several monitors failing
TLS validation against self-signed/internal-IP certs) — confirmed resolved
by the operator independently of this change. This log rotation fix is
not causally tied to that incident; it's still worth merging as
standalone disk/IO hygiene given the file size and shared-storage pressure.
Why this needed its own design (not just "add a logrotate conf")
traefik_reverse-proxyruns Swarmmode: global— one Traefikinstance on each of docker-1/2/3, all three writing to the same
physical file via the shared CephFS mount
/volume1/docker/traefik -> /traefik. That rules out standard signal-based rotation (no reopenhandler in Traefik, and no clean way to signal 3 independent per-node
containers in lockstep) and rules out running logrotate on just one node
(silent SPOF if that node is down).
Solution:
copytruncate— all three Traefik processes keep appending to thesame inode, no signaling required.
flock+ sharedlogrotate state file (both on the same CephFS mount) — whichever node's
cron fires first does the rotation; no single node is a SPOF.
fast under bursts, so a
dailyrule alone wouldn't bound it well.Full rationale in
traefik/logrotate/README.md.What's included
traefik/logrotate/traefik-logs.conf— logrotate configtraefik/logrotate/traefik-logrotate.sh— cron-invoked wrapper (flock + shared state)traefik/logrotate/install.sh— one-time per-node installertraefik/logrotate/README.md— full rationale, install steps, and manual bootstrap instructions for shrinking the existing 15.5GB file (deliberately not automated — needs disk-headroom awareness given/volume1/docker-rootis at 82%+)What's NOT included (by design)
traefik.yaml/ no stack redeploy — this is host-levelcron/logrotate config that
stack-deploy.shhas no reach into (it onlytouches Swarm services). Zero risk to the running
traefikstack.--log.level=DEBUG— flagged in the README as relatedfollow-up debt, but lowering it means a real modification + redeploy of
a high-blast-radius stack, so it's out of scope here.
steps in the README; needs a disk-space check first (~14G free at last
check,
copytruncateneeds roughly the source file's size in headroom).Rollout (manual, once per node, not via stack-deploy.sh)
Risk
Low — no Swarm service touched, no
traefik.yamlchange, additive-onlyhost config. Worst case if misconfigured: rotation doesn't happen (status
quo), not a routing outage.