From a2308b76f80240971e29d76af1dc380622d47fbe Mon Sep 17 00:00:00 2001 From: Bot Date: Sat, 12 Sep 2026 00:21:36 -0700 Subject: [PATCH] traefik: fix corrupted README (was double base64-encoded) --- traefik/logrotate/README.md | 121 +++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/traefik/logrotate/README.md b/traefik/logrotate/README.md index d8c3b75..ce5055b 100644 --- a/traefik/logrotate/README.md +++ b/traefik/logrotate/README.md @@ -1 +1,120 @@ -IyBUcmFlZmlrIExvZyBSb3RhdGlvbgoKIyMgV2h5IHRoaXMgZXhpc3RzCgpgdHJhZWZpay95YW1sYCBydW5zIFRyYWVmaWsgd2l0aDoKLSBgLS1hY2Nlc3Nsb2cuZmlsZVBhdGg9L3RyYWVmaWsvbG9ncy9hY2Nlc3MubG9nYAotIGAtLWxvZy5maWxlUGF0aD0vdHJhZWZpay9sb2dzL3RyYWVmaWsubG9nYAotIGAtLWxvZy5sZXZlbD1EQlVHYCAoc2ljIC0gYWN0dWFsbHkgYERFQlVHYCkKCk5laXRoZXIgZmlsZSBoYXMgYW55IGJ1aWx0LWluIHJvdGF0aW9uIC0tIFRyYWVmaWsgaGFzIG5vIG5hdGl2ZSByb3RhdGUtb24tc2l6ZQotIG5vciBhIFNJR1VTUjEvcmVvcGVuIGhhbmRsZXIuIERvY2tlcidzIGBqc29uLWZpbGVgIGxvZy1kcml2ZXIgcm90YXRpb24KKGBtYXgtc2l6ZWAvYG1heC1maWxlYCkgb25seSBhcHBsaWVzIHRvIHN0ZG91dCwgbm90IHRvIGZpbGVzIFRyYWVmaWsgd3JpdGVzCmRpcmVjdGx5IHZpYSBgLS1hY2Nlc3Nsb2cuZmlsZVBhdGhgL2AtLWxvZy5maWxlUGF0aGAuIFJlc3VsdDogYGFjY2Vzcy5sb2dgIGdyZXcKdG8gKip+MTUuNUdCIHVucm90YXRlZCoqIGJlZm9yZSB0aGlzIHdhcyBjYXVnaHQsIG9uIGEgQ2VwaEZTIHZvbHVtZQooYC92b2x1bWUxL2RvY2tlci1yb290YCkgYWxyZWFkeSBhdCA4Mi04NCUgdXNlZC5Cb3RoIHRoZSBkaXNrIHByZXNzdXJlIGFuZAp0aGUgb25nb2luZyB3cml0ZSBsYXRlbmN5IG9mIGFwcGVuZGluZyB0byBhIDE1R0IgZmlsZSBvbiBhIG5ldHdvcmsgZmlsZXN5c3RlbQpvbiBFVkVSWSByZXF1ZXN0IHRocm91Z2ggVHJhZWZpayB3ZXJlIGZsYWdnZWQgYXMgYSByZWFsIHJpc2sgZmFjdG9yIGR1cmluZwp0cm91Ymxlc2hvb3RpbmcgKFVwdGltZSBLdW1hIFdlYlNvY2tldCBmbGFwcGluZyBpbnZlc3RpZ2F0aW9uLCBTZXAgMjAyNikuCgojIyBBcmNoaXRlY3R1cmUgY29uc3RyYWludCB0aGlzIGRlc2lnbiBhY2NvdW50cyBmb3IKCmB0cmFlZmlrX3JldmVyc2UtcHJveHlgIHJ1bnMgU3dhcm0gKmBtb2RlOiBnbG9iYWxgKiAtLSBvbmUgaW5zdGFuY2Ugb24gKiplYWNoKioKb2YgZG9ja2VyLTEsIGRvY2tlci0yLCBkb2NrZXItMy4gQWxsIHRocmVlIHdyaXRlIHRvIHRoZSAqKnNhbWUgcGh5c2ljYWwgZmlsZSoqCnZpYSB0aGUgc2hhcmVkIENlcGhGUyBiaW5kIG1vdW50IGAvdm9sdW1lMS9kb2NrZXIvdHJhZWZpayAtPiAvdHJhZWZpa2AgKGlkZW50aWNhbAptb3VudCwgdmlzaWJsZSBpZGVudGljYWxseSBmcm9tIGFueSBub2RlKS4gVGhhdCBydWxlcyBvdXQ6CgotICoqU2lnbmFsLWJhc2VkIHJvdGF0aW9uKiogKGNsYXNzaWMgYGNyZWF0ZWAgKyBgcG9zdHJvdGF0ZWAgc2VuZGluZyBTSUdVU1IxKToKICBUcmFlZmlrIGRvZXNuJ3QgaW1wbGVtZW50IGEgcmVvcGVuIHNpZ25hbCwgYW5kIGV2ZW4gaWYgaXQgZGlkLCB5b3UnZCBuZWVkIHRvCiAgc2lnbmFsIDMgc2VwYXJhdGUgcGVyLW5vZGUgY29udGFpbmVycyBpbiBsb2Nrc3RlcC4KLSAqKlJ1bm5pbmcgbG9ncm90YXRlIG9uIGp1c3Qgb25lIG5vZGUqKjogd29ya3MgdW50aWwgdGhhdCBub2RlIGlzIGRvd24sIHRoZW4KICByb3RhdGlvbiBzaWxlbnRseSBzdG9wcyB3aXRoIG5vIGFsZXJ0LgoKU28gdGhpcyBzZXR1cCB1c2VzOgoxLiAqKmBjb3B5dHJ1bmNhdGVgKiogKHNlZSBgdHJhZWZpay1sb2dzLmNvbmZgKSAtLSBhbGwgdGhyZWUgVHJhZWZpayBwcm9jZXNzZXMKICAga2VlcCB3cml0aW5nIHRvIHRoZSBzYW1lIGlub2RlLCBubyBzaWduYWxpbmcgbmVlZGVkLiBUcmFkZW9mZjogYSBmZXcgbG9nCiAgIGxpbmVzIHdyaXR0ZW4gaW4gdGhlIGV4YWN0IGNvcHkvdHJ1bmNhdGUgaW5zdGFudCBjYW4gYmUgbG9zdCAtLSBmaW5lIGZvcgogICBkaWFnbm9zdGljIGxvZ3MuCjIuICoqQ3JvbiBvbiBhbGwgdGhyZWUgbm9kZXMqKiwgY29vcmRpbmF0ZWQgdmlhIGEgc2hhcmVkIGBmbG9ja2AgKyBzaGFyZWQKICAgbG9ncm90YXRlIHN0YXRlIGZpbGUsIGJvdGggYWxzbyBvbiB0aGUgQ2VwaEZTIG1vdW50IChzZWUKICAgYHRyYWVmaWstbG9ncm90YXRlLnNoYCkuIFdoaWNoZXZlciBub2RlJ3MgY3JvbiBmaXJlcyBmaXJzdCBncmFicyB0aGUgbG9jaywKICAgcm90YXRlcyBpZiBkdWUsIGFuZCB1cGRhdGVzIHRoZSBzaGFyZWQgc3RhdGUgc28gdGhlIG90aGVyIHR3byBub2Rlcydcm9uCiAgIHJ1bnMgc2VlIGl0J3MgYWxyZWFkeSBkb25lLiBObyBzaW5nbGUgbm9kZSBpcyBhIHJvdGF0aW9uIFNQT0YuCjMuICoqU2l6ZS10cmlnZ2VyZWQqKiAoYHNpemUgMjUwTWApIHJhdGhlciB0aGFuIGNhbGVuZGFyLXRyaWdnZXJlZCAoYGRhaWx5YCkgLS0KICAgdGhpcyBmaWxlIGNhbiBncm93IGZhc3QgdW5kZXIgYnVyc3RzIChzZWUgaW5jaWRlbnQgYmFja2dyb3VuZCBhYm92ZSk7IGEKICAgcHVyZWx5IGRhaWx5IGludGVydmFsIHdvdWxkIHN0aWxsIGxldCBpdCBiYWxsb29uIGJldHdlZW4gcnVucy4gQ3JvbiBjaGVja3MKICAgZXZlcnkgMTUgbWludXRlcywgc28gaXQgY2Fubm90IGdyb3cgbXVjaCBwYXN0IHRoZSAyNTBNIHRocmVzaG9sZCBpbiBwcmFjdGljZS4KCiMjIEluc3RhbGwgKG9uZS10aW1lLCBwZXIgbm9kZSkKCk11c3QgcnVuIG9uICoqYWxsIHRocmVlKiogZG9ja2VyIExYQ3MgLS0gdGhpcyBpcyBob3N0LWxldmVsIGNyb24vbG9ncm90YXRlCmNvbmZpZywgbm90IHNvbWV0aGluZyBgc3RhY2stZGVwbG95LnNoYCBjYW4gcmVhY2ggKGl0IG9ubHkgdG91Y2hlcyBTd2FybQpzZXJ2aWNlcywgbm90IGhvc3QgY3JvbiBqb2JzKS4KCmBgYGJhc2gKIyBPbiBlYWNoIG9mIGRvY2tlci0xICgxOTIuMTY4LjQuMzEpLCBkb2NrZXItMiAoMTkyLjE2OC40LjMyKSwgZG9ja2VyLTMgKDE5Mi4xNjguNC4zMyk6CnNzaCByb290QDwxOTIuMTY4LjQuMzF8LjMyfC4zMz4KY2QgL3ZvbHVtZTEvZG9ja2VyL2NvbXBvc2UtZmlsZXMKYmFzaCBkZXBsb3kvZ2l0LWd1YXJkLnNoICAgICAgICAgIyBjb25maXJtIHN5bmMgZmlyc3QsIGFzIGFsd2F5cwpzdWRvIGJhc2ggdHJhZWZpay9sb2dyb3RhdGUvaW5zdGFsbC5zaAogYGBgCgpWZXJpZnkgb24gZWFjaCBub2RlOgpgYGBiYXNoCmNhdCAvZXRjL2Nyb24uZC90cmFlZmlrLWxvZ3JvdGF0ZQpsb2dyb3RhdGUgLWQgL2V0Yy9sb2dyb3RhdGUuZC90cmFlZmlrLWxvZ3MgICAjIGRyeS1ydW4sIGNvbmZpcm1zIHN5bnRheAogYGBgCgojIyBCb290c3RyYXBwaW5nIC0tIHNocmlua2luZyB0aGUgKmVYaXN0aW5nKiBvdmVyc2l6ZWQgbG9nIGZpbGUKCioqTm90IGRvbmUgYXV0b21hdGljYWxseSBieSBgaW5zdGFsbC5zaGAuKiogVGhlIG5ldyBzaXplLXRyaWdnZXJlZCBjb25maWcgb25seQpwcmV2ZW50cyAqZnV0dXJlKiB1bmJvdW5kZWQgZ3Jvd3RoIC0tIGl0IHdvbid0IHRvdWNoIHRoZSBjdXJyZW50IDE1LjVHQiBmaWxlCnVudGlsIHRoZSBuZXh0IHRpbWUgaXQgY3Jvc3NlcyAyNTBNIChpLmUuIG5ldmVyLCBzaW5jZSBpdCdzIGFscmVhZHkgd2VsbCBwYXN0CnRoYXQgYW5kIGxvZ3JvdGF0ZSBvbmx5IGFjdHMgb24gY3Jvc3NpbmcgdGhlIHRocmVzaG9sZCBnb2luZyBmb3J3YXJkIGZyb20gaXRzCnJlY29yZGVkIHNpemUgYXQgbGFzdCBjaGVjaykuCgpgY29weXRydW5jYXRlYCBhbHdheXMgY29waWVzIHRoZSBmdWxsIGN1cnJlbnQgZmlsZSBiZWZvcmUgdHJ1bmNhdGluZyBpdCAtLSB0aGF0J3MKaW5oZXJlbnQgdG8gaG93IGl0IHdvcmtzLCBub3QgYSBidWcuIEZvcmNpbmcgYSByb3RhdGlvbiBvZiB0aGUgY3VycmVudCAxNS41R0IKZmlsZSB3b3VsZCBtb21lbnRhcmlseSBuZWVkIHJvdWdobHkgYW5vdGhlciAxNS1HQi1zaXplZCBjaHVuayBvZiBmcmVlIHNwYWNlIC0tCmFuZCBgL3ZvbHVtZTEvZG9ja2VyLXJvb3RgIG9ubHkgaGFkICoqfjE0RyBmcmVlKiogYXQgbGFzdCBjaGVjay4gRG9pbmcgdGhpcwpibGluZGx5IGNvdWxkIHRpcCB0aGUgdm9sdW1lIHRvIDEwMCUgbWlkLW9wZXJhdGlvbi4KCioqUmVjb21tZW5kZWQgbWFudWFsIHN0ZXAgKG9wZXJhdG9yLXJ1biwgb24gYW55IG9uZSBub2RlIC0tIGl0J3MgdGhlIHNhbWUKQ2VwaEZTIGZpbGUgZnJvbSBhbGwgdGhyZWUpKio6IHRoZXNlIGFyZSBkaWFnbm9zdGljIGFjY2Vzcy9kZWJ1ZyBsb2dzLCBub3Qgc29tZXRoaW5nCndvcnRoIHByZXNlcnZpbmcgaW4gZnVsbCwgc28ganVzdCB0cnVuY2F0ZSBkaXJlY3RseSByYXRoZXIgdGhhbiBjb21wcmVzcy10aGVuLQp0cnVuY2F0ZTogCgpgYGBiYXNoCiMgT3B0aW9uYWw6IGtlZXAgYSBzbWFsbCB0YWlsIHNhbXBsZSBmb3IgcmVmZXJlbmNlIGJlZm9yZSB0cnVuY2F0aW5nCnRhaWwgLWMgNTAwMDAwMDAgL3ZvbHVtZTEvZG9ja2VyL3RyYWVmaWsvbG9ncy9hY2Nlc3MubG9nID4gXAogIC92b2x1bWUxL2RvY2tlci90cmFlZmlrLWFjY2Vzcy1sb2ctYXJjaGl2ZS0kKGRhdGUgKyVZJW0lZCkubG9nCgojIFRoZW4gdHJ1bmNhdGUgaW4gcGxhY2UgKHNhZmUgLS0gVHJhZWZpaydzIGV4aXN0aW5nIGZpbGUgaGFuZGxlcyBvbiBhbGwgMwojIG5vZGVzIHN0YXkgdmFsaWQsIHNhbWUgYXMgY29weXRydW5jYXRlJ3Mgb3duIG1lY2hhbmlzbSk6CjogPiAvdm9sdW1lMS9kb2NrZXIvdHJhZWZpay9sb2dzL2FjY2Vzcy5sb2cKOiA+IC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MvdHJhZWZpay5sb2cKCiMgQ29uZmlybToKZGYgLWggL3ZvbHVtZTEvZG9ja2VyLXJvb3QKbHMgLWxhIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MvCmBgYAoKQWZ0ZXIgdGhpcyBvbmUtdGltZSBib290c3RyYXAsIHRoZSBjcm9uK2xvZ3JvdGF0ZSBzZXR1cCBrZWVwcyBpdCBib3VuZGVkCihyb3RhdGVzIGF0IDI1ME0sIGtlZXBzIDQ4IGNvbXByZXNzZWQgZ2VuZXJhdGlvbnMsIHBydW5lcyBhbnl0aGluZyBvdmVyIDE0CmRheXMgb2xkKSBnb2luZyBmb3J3YXJkIHdpdGhvdXQgbmVlZGluZyBhbnkgZnVydGhlciBtYW51YWwgaW50ZXJ2ZW50aW9uLgoKIyMgUmVsYXRlZCwgTk9UIGluY2x1ZGVkIGluIHRoaXMgY2hhbmdlIChmb2xsb3ctdXAgdG8gY29uc2lkZXIgc2VwYXJhdGVseSkKCmB0cmFlZmlrLnlhbWxgIGN1cnJlbnRseSBydW5zIGAtLWxvZy5sZXZlbD1EQlVHYCAtLSB2ZXJib3NlIGRlYnVnIGxvZ2dpbmcgaW4KcHJvZHVjdGlvbiwgd2hpY2ggaXMgYSBtZWFuaW5nZnVsIGNvbnRyaWJ1dG9yIHRvIGhvdyBmYXN0IHRoZXNlIGZpbGVzIGdyb3cuCkxvd2VyaW5nIHRvIGBJTkZPYCB3b3VsZCByZWR1Y2Ugdm9sdW1lIHNpZ25pZmljYW50bHkgYnV0IHJlcXVpcmVzIGEgcmVhbAptb2RpZmljYXRpb24gK3JlZGVwbG95IG9mIHRoZSBoaWdoLWJsYXN0LXJhZGl1cyBgdHJhZWZpa2Agc3RhY2sgKGFsbCBIVFRQL0hUVFBTCApyb3V0aW5nIGRlcGVuZHMgb24gaXQpLCBzbyBpdCdzIGludGVudGlvbmFsbHkgbGVmdCBvdXQgb2YgdGhpcyBQUiBhbmQgc2hvdWxkCmJlIGl0cyBvd24gcmV2aWV3ZWQgY2hhbmdlIGlmIHdhbnRlZC4= \ No newline at end of file +# Traefik Log Rotation + +## Why this exists + +`traefik/traefik.yaml` runs Traefik with: +- `--accesslog.filePath=/traefik/logs/access.log` +- `--log.filePath=/traefik/logs/traefik.log` +- `--log.level=DEBUG` + +Neither file has any built-in rotation -- Traefik has no native rotate-on-size +nor a SIGUSR1/reopen handler. Docker's `json-file` log-driver rotation +(`max-size`/`max-file`) only applies to stdout, not to files Traefik writes +directly via `--accesslog.filePath`/`--log.filePath`. Result: `access.log` grew +to **~15.5GB unrotated** before this was caught, on a CephFS volume +(`/volume1/docker-root`) already at 82-84% used. Both the disk pressure and +the ongoing write latency of appending to a 15GB file on a network filesystem +on every request through Traefik were flagged as a real risk factor during +troubleshooting (Uptime Kuma WebSocket flapping investigation, Sep 2026). + +**Note:** during that investigation, the actual root cause of the WebSocket +flapping turned out to be Uptime Kuma monitor misconfigurations (a Postgres +monitor throwing a null-reference error, and several monitors failing TLS +validation against self-signed/internal-IP certs) -- not this log file. This +rotation fix is still worth doing as general disk/IO hygiene, just not +causally tied to that incident. + +## Architecture constraint this design accounts for + +`traefik_reverse-proxy` runs Swarm *`mode: global`* -- one instance on **each** +of docker-1, docker-2, docker-3. All three write to the **same physical file** +via the shared CephFS bind mount `/volume1/docker/traefik -> /traefik` +(identical mount, visible identically from any node). That rules out: + +- **Signal-based rotation** (classic `create` + `postrotate` sending SIGUSR1): + Traefik doesn't implement a reopen signal, and even if it did, you'd need to + signal 3 separate per-node containers in lockstep. +- **Running logrotate on just one node**: works until that node is down, then + rotation silently stops with no alert. + +So this setup uses: +1. **`copytruncate`** (see `traefik-logs.conf`) -- all three Traefik processes + keep writing to the same inode, no signaling needed. Tradeoff: a few log + lines written in the exact copy/truncate instant can be lost -- fine for + diagnostic logs. +2. **Cron on all three nodes**, coordinated via a shared `flock` + shared + logrotate state file, both also on the CephFS mount (see + `traefik-logrotate.sh`). Whichever node's cron fires first grabs the lock, + rotates if due, and updates the shared state so the other two nodes' cron + runs see it's already done. No single node is a rotation SPOF. +3. **Size-triggered** (`size 250M`) rather than calendar-triggered (`daily`) -- + this file can grow fast under bursts (see incident background above); a + purely daily interval would still let it balloon between runs. Cron checks + every 15 minutes, so it cannot grow much past the 250M threshold in practice. + +## Install (one-time, per node) + +Must run on **all three** docker LXCs -- this is host-level cron/logrotate +config, not something `stack-deploy.sh` can reach (it only touches Swarm +services, not host cron jobs). + +```bash +# On each of docker-1 (192.168.4.31), docker-2 (192.168.4.32), docker-3 (192.168.4.33): +ssh root@<192.168.4.31|.32|.33> +cd /volume1/docker/compose-files +bash deploy/git-guard.sh # confirm sync first, as always +sudo bash traefik/logrotate/install.sh +``` + +Verify on each node: +```bash +cat /etc/cron.d/traefik-logrotate +logrotate -d /etc/logrotate.d/traefik-logs # dry-run, confirms syntax +``` + +## Bootstrapping -- shrinking the *existing* oversized log file + +**Not done automatically by `install.sh`.** The new size-triggered config only +prevents *future* unbounded growth -- it won't touch the current 15.5GB file +until the next time it crosses 250M (i.e. never, since it's already well past +that and logrotate only acts on crossing the threshold going forward from its +recorded size at last check). + +`copytruncate` always copies the full current file before truncating it -- +that's inherent to how it works, not a bug. Forcing a rotation of the current +15.5GB file would momentarily need roughly another 15-GB-sized chunk of free +space -- and `/volume1/docker-root` only had **~14G free** at last check. +Doing this blindly could tip the volume to 100% mid-operation. + +**Recommended manual step (operator-run, on any one node -- it's the same +CephFS file from all three)**: these are diagnostic access/debug logs, not +something worth preserving in full, so just truncate directly rather than +compress-then-truncate: + +```bash +# Optional: keep a small tail sample for reference before truncating +tail -c 50000000 /volume1/docker/traefik/logs/access.log > \ + /volume1/docker/traefik-access-log-archive-$(date +%Y%m%d).log + +# Then truncate in place (safe -- Traefik's existing file handles on all 3 +# nodes stay valid, same as copytruncate's own mechanism): +: > /volume1/docker/traefik/logs/access.log +: > /volume1/docker/traefik/logs/traefik.log + +# Confirm: +df -h /volume1/docker-root +ls -la /volume1/docker/traefik/logs/ +``` + +After this one-time bootstrap, the cron+logrotate setup keeps it bounded +(rotates at 250M, keeps 48 compressed generations, prunes anything over 14 +days old) going forward without needing any further manual intervention. + +## Related, NOT included in this change (follow-up to consider separately) + +`traefik.yaml` currently runs `--log.level=DEBUG` -- verbose debug logging in +production, which is a meaningful contributor to how fast these files grow. +Lowering to `INFO` would reduce volume significantly but requires a real +modification + redeploy of the high-blast-radius `traefik` stack (all +HTTP/HTTPS routing depends on it), so it's intentionally left out of this PR +and should be its own reviewed change if wanted.