From e27d2f6ac2939990081bfe58c713a9aad928b6b8 Mon Sep 17 00:00:00 2001 From: Bot Date: Fri, 11 Sep 2026 23:29:00 -0700 Subject: [PATCH 1/9] traefik: add logrotate config for access/traefik logs Traefik has no built-in log rotation and no SIGUSR1/reopen handling. --accesslog.filePath and --log.filePath (traefik.yaml) write directly to /volume1/docker/traefik/logs, which had grown to 15.5GB unrotated, contributing to disk pressure (docker-2 CephFS at 82%) and adding write latency to every request through Traefik. --- traefik/logrotate/traefik-logs.conf | 1 + 1 file changed, 1 insertion(+) create mode 100644 traefik/logrotate/traefik-logs.conf diff --git a/traefik/logrotate/traefik-logs.conf b/traefik/logrotate/traefik-logs.conf new file mode 100644 index 0000000..ac6add8 --- /dev/null +++ b/traefik/logrotate/traefik-logs.conf @@ -0,0 +1 @@ +IyBUcmFlZmlrIGFjY2Vzcy9lcnJvciBsb2cgcm90YXRpb24KIwojIENPTlRFWFQ6IHRyYWVmaWtfcmV2ZXJzZS1wcm94eSBydW5zIGluIFN3YXJtIGBtb2RlOiBnbG9iYWxgICh0cmFlZmlrLnlhbWwpLAojIG1lYW5pbmcgb25lIGluc3RhbmNlIHJ1bnMgb24gRUFDSCBvZiBkb2NrZXItMS9kb2NrZXItMi9kb2NrZXItMy4gQWxsIHRocmVlCiMgd3JpdGUgdG8gdGhlIFNBTUUgcGh5c2ljYWwgZmlsZSB2aWEgdGhlIHNoYXJlZCBDZXBoRlMgYmluZCBtb3VudAojIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MgLT4gL3RyYWVmaWsvbG9ncyAoaWRlbnRpY2FsIHBhdGggZnJvbSBhbnkgbm9kZSAtLQojIHNlZSBpbmZyYSBjb250ZXh0OiAvdm9sdW1lMS9kb2NrZXIgaXMgYSBzaGFyZWQgQ2VwaEZTIG1vdW50KS4KIwojIGBjb3B5dHJ1bmNhdGVgIGlzIFJFUVVJUkVEIGhlcmUgKG5vdCB0aGUgZGVmYXVsdCBjcmVhdGUrc2lnbmFsIGFwcHJvYWNoKToKIyBUcmFlZmlrIGhhcyBubyBTSUdVU1IxL1NJR0hVUCAicmVvcGVuIGxvZyBmaWxlIiBoYW5kbGluZywgYW5kIGV2ZW4gaWYgaXQKIyBkaWQsIGNvb3JkaW5hdGluZyBhIHJlb3BlbiBzaWduYWwgYWNyb3NzIDMgaW5kZXBlbmRlbnQgcGVyLW5vZGUgY29udGFpbmVycwojIHdyaXRpbmcgdG8gb25lIHNoYXJlZCBpbm9kZSBpcyB1bm5lY2Vzc2FyeSBjb21wbGV4aXR5LiBjb3B5dHJ1bmNhdGUga2VlcHMKIyBldmVyeSB3cml0ZXIncyBleGlzdGluZyBmaWxlIGRlc2NyaXB0b3IgdmFsaWQgKHRydW5jYXRlcyBpbiBwbGFjZSkgc28gYWxsCiMgdGhyZWUgVHJhZWZpayBwcm9jZXNzZXMga2VlcCBhcHBlbmRpbmcgdG8gdGhlIHNhbWUgaW5vZGUgd2l0aCB6ZXJvCiMgc2lnbmFsaW5nLiBUcmFkZW9mZjogYSBoYW5kZnVsIG9mIGxvZyBsaW5lcyB3cml0dGVuIGluIHRoZSBleGFjdAojIGNvcHkvdHJ1bmNhdGUgaW5zdGFudCBjYW4gYmUgbG9zdCAtLSBhY2NlcHRhYmxlIGZvciBkaWFnbm9zdGljIGFjY2Vzcy9lcnJvcgojIGxvZ3MsIG5vdCB1c2VkIGZvciBhbnl0aGluZyB0cmFuc2FjdGlvbmFsLgojCiMgU2l6ZS10cmlnZ2VyZWQgKG5vdCBjYWxlbmRhci10cmlnZ2VyZWQpIG9uIHB1cnBvc2U6IHRoaXMgZmlsZSBjYW4gZ3JvdyBmYXN0CiMgdW5kZXIgYnVyc3RzIChzZWUgaW5jaWRlbnQgdGhhdCBwcm9tcHRlZCB0aGlzIC0tIDE1LjVHQiBhY2N1bXVsYXRlZCB3aXRoCiMgLS1sb2cubGV2ZWw9REVCVUcgc2V0KS4gYHNpemVgIGlzIGNoZWNrZWQgZXZlcnkgdGltZSB0aGUgd3JhcHBlciBzY3JpcHQgcnVucwojIChjcm9uLCBldmVyeSAxNSBtaW51dGVzIC0tIHNlZSBpbnN0YWxsLnNoKSwgc28gaXQgY2Fubm90IGJhbGxvb24gdW5ib3VuZGVkCiMgYmV0d2VlbiBjaGVja3MgdGhlIHdheSBhIHBsYWluIGBkYWlseWAgaW50ZXJ2YWwgd291bGQuCiMKIyBJbnN0YWxsZWQgdmlhIGluc3RhbGwuc2ggb24gQUxMIFRIUkVFIGRvY2tlciBMWENzIChkb2NrZXItMSwgZG9ja2VyLTIsCiMgZG9ja2VyLTMpIC0tIHNlZSBSRUFETUUubWQuIFRoaXMgaXMgYSBIT1NULWxldmVsIGNyb24vbG9ncm90YXRlIGNvbmZpZywKIyBvdXRzaWRlIHRoZSBXb29kcGVja2VyL3N0YWNrLWRlcGxveS5zaCBwaXBlbGluZSAoZG9ja2VyIHN0YWNrIGRlcGxveSBoYXMgbm8KIyBtZWNoYW5pc20gdG8gdG91Y2ggaG9zdCBjcm9uKSwgc28gaXQgbXVzdCBiZSBhcHBsaWVkIG1hbnVhbGx5IG9uY2UgcGVyIG5vZGUsCiMgbm90IHZpYSBhIHN0YWNrIHJlZGVwbG95LgoKL3ZvbHVtZTEvZG9ja2VyL3RyYWVmaWsvbG9ncy8qLmxvZyB7CiAgICBzaXplIDI1ME0KICAgIHJvdGF0ZSA0OAogICAgbWF4YWdlIDE0CiAgICBjb21wcmVzcwogICAgZGVsYXljb21wcmVzcwogICAgbWlzc2luZ29rCiAgICBub3RpZmVtcHR5CiAgICBjb3B5dHJ1bmNhdGUKICAgIGRhdGVleHQKICAgIGRhdGVmb3JtYXQgLSVZJW0lZC0lSCVNJVMKICAgIHN1IHJvb3Qgcm9vdAp9Cg== \ No newline at end of file From 4f4c5a467b55d32319b34f8266a2ee3b94d916b8 Mon Sep 17 00:00:00 2001 From: Bot Date: Fri, 11 Sep 2026 23:29:00 -0700 Subject: [PATCH 2/9] traefik: add cron wrapper for coordinated multi-node log rotation reverse-proxy runs Swarm mode:global (one instance per node), all writing to the same CephFS file. This wrapper uses a shared flock + shared logrotate state file (both also on the CephFS mount) so cron on docker-1/2/3 can run independently without racing or double-rotating. --- traefik/logrotate/traefik-logrotate.sh | 1 + 1 file changed, 1 insertion(+) create mode 100644 traefik/logrotate/traefik-logrotate.sh diff --git a/traefik/logrotate/traefik-logrotate.sh b/traefik/logrotate/traefik-logrotate.sh new file mode 100644 index 0000000..c27a8dc --- /dev/null +++ b/traefik/logrotate/traefik-logrotate.sh @@ -0,0 +1 @@ +IyEvdXNyL2Jpbi9lbnYgYmFzaAojIFdyYXBwZXIgaW52b2tlZCBieSBjcm9uIG9uIGRvY2tlci0xL2RvY2tlci0yL2RvY2tlci0zIHRvIHJvdGF0ZSB0aGUgc2hhcmVkCiMgVHJhZWZpayBhY2Nlc3MvZXJyb3IgbG9ncyAoc2VlIHRyYWVmaWstbG9ncy5jb25mIGZvciB0aGUgImZ1bGwgd2h5IikuCiMKIyBCZWNhdXNlIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MgaXMgdGhlIFNBTUUgcGh5c2ljYWwgQ2VwaEZTIHBhdGggb24KIyBhbGwgdGhyZWUgbm9kZXMsIGFuZCBjcm9uIG9uIGFsbCB0aHJlZSBub2RlcyBydW5zIHRoaXMgaW5kZXBlbmRlbnRseSwgd2UKIyB1c2UgYSBzaGFyZWQgZmxvY2sgKGFsc28gb24gdGhlIENlcGhGUyBtb3VudCwgc28gaXQncyB2aXNpYmxlIGNsdXN0ZXItd2lkZSkKIyB0byBndWFyYW50ZWUgb25seSBvbmUgbm9kZSBhY3R1YWxseSBleGVjdXRlcyBsb2dyb3RhdGUgYXQgYSB0aW1lLCBhbmQgYQojIFNIQVJFRCBzdGF0ZSBmaWxlIHNvIHdoaWNoZXZlciBub2RlIHJ1bnMgaXQga25vd3MgdGhlIHRydWUgbGFzdC1yb3RhdGVkCiMgdGltZSByZWdhcmRsZXNzIG9mIHdoaWNoIG5vZGUgcm90YXRlZCBpdCBsYXN0LiBJZiBhIG5vZGUgaXMgZG93biwgdGhlCiMgb3RoZXIgdHdvIHN0aWxsIGNvdmVyIHRoZSBzY2hlZHVsZSAtLSBub25lIG9mIHRoaXMgcmVsaWVzIG9uIGEgc3BlY2lmaWMKIyBub2RlIGJlaW5nIHVwLgpzZXQgLWV1byBwaXBlZmFpbAoKTE9DS19ESVI9Ii92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3JvdGF0ZS1zdGF0ZSIKTE9DS19GSUxFPSIkeyBMT0NLX0RJUn0vcm90YXRlLmxvY2siCkxPQ0tfRklMRT0iJHtMT0NLX0RJUn0vcm90YXRlLmxvY2siClNUQVRFX0ZJTEU9IiR7TE9DS19ESVJ9L3N0YXR1cyIKQ09ORl9GSUxFPSIvZXRjL2xvZ3JvdGF0ZS5kL3RyYWVmaWstbG9ncyIKCm1rZGlyIC1wICIkTE9DS19ESVIiCnRvdWNoICIkU1RBVEVfRklMRSIKCmV4ZWMgMjAwPiIkTE9DS19GSUxFIgppZiAhIGZsb2NrIC1uIDIwMDsgdGhlbgogICMgQW5vdGhlciBub2RlIGFscmVhZHkgaG9sZHMgdGhlIGxvY2sgdGhpcyBjeWNsZSAtLSBub3JtYWwsIG5vdCBhbiBlcnJvci4KICBleGl0IDAKZmkKCi91c3Ivc2Jpbi9sb2dyb3RhdGUgLXMgIiRTVEFURV9GSUxFIiAiJENPTkZfRklMRSI= \ No newline at end of file From 6ddb31c594142df6105652189c6c07d95d79edfe Mon Sep 17 00:00:00 2001 From: Bot Date: Fri, 11 Sep 2026 23:29:01 -0700 Subject: [PATCH 3/9] traefik: add one-time logrotate installer (run on each node) --- traefik/logrotate/install.sh | 1 + 1 file changed, 1 insertion(+) create mode 100644 traefik/logrotate/install.sh diff --git a/traefik/logrotate/install.sh b/traefik/logrotate/install.sh new file mode 100644 index 0000000..27f0f6a --- /dev/null +++ b/traefik/logrotate/install.sh @@ -0,0 +1 @@ +IyEvdXNyL2Jpbi9lbnYgYmFzaAojIE9uZS10aW1lIGluc3RhbGxlciBmb3IgVHJhZWZpayBsb2cgcm90YXRpb24uCiMKIyBNVVNUIGJlIHJ1biBtYW51YWxseSwgb25jZSwgb24gRUFDSCBvZiBkb2NrZXItMSwgZG9ja2VyLTIsIGRvY2tlci0zLgojIFRoaXMgaXMgaW50ZW50aW9uYWxseSBOT1QgcGFydCBvZiBzdGFjay1kZXBsb3kuc2ggLyB0aGUgV29vZHBlY2tlcgojIHBpcGVsaW5lOiBpdCBpbnN0YWxscyBhIGhvc3QtbGV2ZWwgY3Jvbi5kIGVudHJ5IGFuZCAvZXRjL2xvZ3JvdGF0ZS5kCiMgY29uZmlnLCBhbmQgYGRvY2tlciBzdGFjayBkZXBsb3lgIGhhcyBubyBtZWNoYW5pc20gdG8gcmVhY2ggb3V0c2lkZSB0aGUKIyBTd2FybS9jb250YWluZXIgYm91bmRhcnkgb250byBob3N0IGNyb24uIFNlZSBSRUFETUUubWQgZm9yIHdoeSB0aGlzIG5lZWRzCiMgdG8gcnVuIG9uIGFsbCB0aHJlZSBub2Rlcy4KIwojIFVzYWdlIChmcm9tIGEgY2hlY2tvdXQgb2YgdGhpcyByZXBvLCBvbiBlYWNoIG5vZGUpOgojICAgc3VkbyBiYXNoIHRyYWVmaWsvbG9ncm90YXRlL2luc3RhbGwuc2gKc2V0IC1ldW8gcGlwZWZhaWwKCmlmIFtbICRFVUlEIC1uZSAwIF1dOyB0aGVuCiAgZWNobyAiUnVuIGFzIHJvb3QgKHN1ZG8pLiIgPiYyCiAgZXhpdCAxCmZpCgpSRVBPX0RJUj0iJChjZCAiJChkaXJuYW1lICIke0JBU0hfU09VUkNFWzBdfSIpIiAmJiBwd2QpIgoKZWNobyAiSW5zdGFsbGluZyBsb2dyb3RhdGUgY29uZmlnLi4uIgppbnN0YWxsIC1tIDA2NDQgIiRSRVBPX0RJUi90cmFlZmlrLWxvZ3MuY29uZiIgL2V0Yy9sb2dyb3RhdGUuZC90cmFlZmlrLWxvZ3MKCmVjaG8gIkluc3RhbGxpbmcgcm90YXRpb24gd3JhcHBlci4uLiIKaW5zdGFsbCAtbSAwNzU1ICIkUkVQT19ESVIvdHJhZWZpay1sb2dyb3RhdGUuc2giIC91c3IvbG9jYWwvc2Jpbi90cmFlZmlrLWxvZ3JvdGF0ZS5zaAoKZWNobyAiSW5zdGFsbGluZyBjcm9uLmQgc2NoZWR1bGUgKGV2ZXJ5IDE1IG1pbnV0ZXMpLi4uIgpjYXQgPiAvZXRjL2Nyb24uZC90cmFlZmlrLWxvZ3JvdGF0ZSA8PCAnRU9GJwojIE1hbmFnZWQgYnkgaG9tZWxhYi9jb21wb3NlLWZpbGVzIHRyYWVmaWsvbG9ncm90YXRlL2luc3RhbGwuc2ggLS0gZG8gbm90CiMgaGFuZC1lZGl0OyB1cGRhdGUgdHJhZWZpay9sb2dyb3RhdGUvKi57Y29uZixzaH0gaW4gR2l0ZWEgYW5kIHJlLXJ1bgojIGluc3RhbGwuc2ggaW5zdGVhZC4KKi8xNSAqICogKiAqIHJvb3QgL3Vzci9sb2NhbC9zYmluL3RyYWVmaWstbG9ncm90YXRlLnNoCkVPRgpjaG1vZCAwNjQ0IC9ldGMvY3Jvbi5kL3RyYWVmaWstbG9ncm90YXRlCgplY2hvICJWZXJpZnlpbmcgbG9ncm90YXRlIGNvbmZpZyBzeW50YXguLi4iCmxvZ3JvdGF0ZSAtZCAvZXRjL2xvZ3JvdGF0ZS5kL3RyYWVmaWstbG9ncwoKZWNobyAiRG9uZS4gRmlyc3Qgcm90YXRpb24gY2hlY2sgcnVucyBvbiB0aGUgbmV4dCBjcm9uIHRpY2sgKHVwIHRvIDE1IG1pbikuIgplY2hvICJUaGlzIG9ubHkgcm90YXRlcyBnb2luZyBmb3J3YXJkIG9uY2UgdGhlIGZpbGUgY3Jvc3NlcyB0aGUgc2l6ZSB0aHJlc2hvbGQuIgplY2hvICJUbyBzaHJpbmsgdGhlIEVYSVNUSU5HIGFscmVhZHktbGFyZ2UgbG9nIGZpbGUsIHNlZSBSRUFETUUubWQgLS0gdGhhdCBpcyIKZWNobyAiYSBzZXBhcmF0ZSwgZGVsaWJlcmF0ZSBtYW51YWwgc3RlcCAobm90IGRvbmUgYnkgdGhpcyBzY3JpcHQpIGJlY2F1c2UgaXQiCmVjaG8gIm5lZWRzIGRpc2sgaGVhZHJvb20gYXdhcmVuZXNzIGZpcnN0LiI= \ No newline at end of file From 409971f7410de6f5a21595f08eb50d3d53fc03f7 Mon Sep 17 00:00:00 2001 From: Bot Date: Fri, 11 Sep 2026 23:29:01 -0700 Subject: [PATCH 4/9] traefik: document log rotation setup, rationale, and bootstrap steps --- traefik/logrotate/README.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 traefik/logrotate/README.md diff --git a/traefik/logrotate/README.md b/traefik/logrotate/README.md new file mode 100644 index 0000000..d8c3b75 --- /dev/null +++ b/traefik/logrotate/README.md @@ -0,0 +1 @@ +IyBUcmFlZmlrIExvZyBSb3RhdGlvbgoKIyMgV2h5IHRoaXMgZXhpc3RzCgpgdHJhZWZpay95YW1sYCBydW5zIFRyYWVmaWsgd2l0aDoKLSBgLS1hY2Nlc3Nsb2cuZmlsZVBhdGg9L3RyYWVmaWsvbG9ncy9hY2Nlc3MubG9nYAotIGAtLWxvZy5maWxlUGF0aD0vdHJhZWZpay9sb2dzL3RyYWVmaWsubG9nYAotIGAtLWxvZy5sZXZlbD1EQlVHYCAoc2ljIC0gYWN0dWFsbHkgYERFQlVHYCkKCk5laXRoZXIgZmlsZSBoYXMgYW55IGJ1aWx0LWluIHJvdGF0aW9uIC0tIFRyYWVmaWsgaGFzIG5vIG5hdGl2ZSByb3RhdGUtb24tc2l6ZQotIG5vciBhIFNJR1VTUjEvcmVvcGVuIGhhbmRsZXIuIERvY2tlcidzIGBqc29uLWZpbGVgIGxvZy1kcml2ZXIgcm90YXRpb24KKGBtYXgtc2l6ZWAvYG1heC1maWxlYCkgb25seSBhcHBsaWVzIHRvIHN0ZG91dCwgbm90IHRvIGZpbGVzIFRyYWVmaWsgd3JpdGVzCmRpcmVjdGx5IHZpYSBgLS1hY2Nlc3Nsb2cuZmlsZVBhdGhgL2AtLWxvZy5maWxlUGF0aGAuIFJlc3VsdDogYGFjY2Vzcy5sb2dgIGdyZXcKdG8gKip+MTUuNUdCIHVucm90YXRlZCoqIGJlZm9yZSB0aGlzIHdhcyBjYXVnaHQsIG9uIGEgQ2VwaEZTIHZvbHVtZQooYC92b2x1bWUxL2RvY2tlci1yb290YCkgYWxyZWFkeSBhdCA4Mi04NCUgdXNlZC5Cb3RoIHRoZSBkaXNrIHByZXNzdXJlIGFuZAp0aGUgb25nb2luZyB3cml0ZSBsYXRlbmN5IG9mIGFwcGVuZGluZyB0byBhIDE1R0IgZmlsZSBvbiBhIG5ldHdvcmsgZmlsZXN5c3RlbQpvbiBFVkVSWSByZXF1ZXN0IHRocm91Z2ggVHJhZWZpayB3ZXJlIGZsYWdnZWQgYXMgYSByZWFsIHJpc2sgZmFjdG9yIGR1cmluZwp0cm91Ymxlc2hvb3RpbmcgKFVwdGltZSBLdW1hIFdlYlNvY2tldCBmbGFwcGluZyBpbnZlc3RpZ2F0aW9uLCBTZXAgMjAyNikuCgojIyBBcmNoaXRlY3R1cmUgY29uc3RyYWludCB0aGlzIGRlc2lnbiBhY2NvdW50cyBmb3IKCmB0cmFlZmlrX3JldmVyc2UtcHJveHlgIHJ1bnMgU3dhcm0gKmBtb2RlOiBnbG9iYWxgKiAtLSBvbmUgaW5zdGFuY2Ugb24gKiplYWNoKioKb2YgZG9ja2VyLTEsIGRvY2tlci0yLCBkb2NrZXItMy4gQWxsIHRocmVlIHdyaXRlIHRvIHRoZSAqKnNhbWUgcGh5c2ljYWwgZmlsZSoqCnZpYSB0aGUgc2hhcmVkIENlcGhGUyBiaW5kIG1vdW50IGAvdm9sdW1lMS9kb2NrZXIvdHJhZWZpayAtPiAvdHJhZWZpa2AgKGlkZW50aWNhbAptb3VudCwgdmlzaWJsZSBpZGVudGljYWxseSBmcm9tIGFueSBub2RlKS4gVGhhdCBydWxlcyBvdXQ6CgotICoqU2lnbmFsLWJhc2VkIHJvdGF0aW9uKiogKGNsYXNzaWMgYGNyZWF0ZWAgKyBgcG9zdHJvdGF0ZWAgc2VuZGluZyBTSUdVU1IxKToKICBUcmFlZmlrIGRvZXNuJ3QgaW1wbGVtZW50IGEgcmVvcGVuIHNpZ25hbCwgYW5kIGV2ZW4gaWYgaXQgZGlkLCB5b3UnZCBuZWVkIHRvCiAgc2lnbmFsIDMgc2VwYXJhdGUgcGVyLW5vZGUgY29udGFpbmVycyBpbiBsb2Nrc3RlcC4KLSAqKlJ1bm5pbmcgbG9ncm90YXRlIG9uIGp1c3Qgb25lIG5vZGUqKjogd29ya3MgdW50aWwgdGhhdCBub2RlIGlzIGRvd24sIHRoZW4KICByb3RhdGlvbiBzaWxlbnRseSBzdG9wcyB3aXRoIG5vIGFsZXJ0LgoKU28gdGhpcyBzZXR1cCB1c2VzOgoxLiAqKmBjb3B5dHJ1bmNhdGVgKiogKHNlZSBgdHJhZWZpay1sb2dzLmNvbmZgKSAtLSBhbGwgdGhyZWUgVHJhZWZpayBwcm9jZXNzZXMKICAga2VlcCB3cml0aW5nIHRvIHRoZSBzYW1lIGlub2RlLCBubyBzaWduYWxpbmcgbmVlZGVkLiBUcmFkZW9mZjogYSBmZXcgbG9nCiAgIGxpbmVzIHdyaXR0ZW4gaW4gdGhlIGV4YWN0IGNvcHkvdHJ1bmNhdGUgaW5zdGFudCBjYW4gYmUgbG9zdCAtLSBmaW5lIGZvcgogICBkaWFnbm9zdGljIGxvZ3MuCjIuICoqQ3JvbiBvbiBhbGwgdGhyZWUgbm9kZXMqKiwgY29vcmRpbmF0ZWQgdmlhIGEgc2hhcmVkIGBmbG9ja2AgKyBzaGFyZWQKICAgbG9ncm90YXRlIHN0YXRlIGZpbGUsIGJvdGggYWxzbyBvbiB0aGUgQ2VwaEZTIG1vdW50IChzZWUKICAgYHRyYWVmaWstbG9ncm90YXRlLnNoYCkuIFdoaWNoZXZlciBub2RlJ3MgY3JvbiBmaXJlcyBmaXJzdCBncmFicyB0aGUgbG9jaywKICAgcm90YXRlcyBpZiBkdWUsIGFuZCB1cGRhdGVzIHRoZSBzaGFyZWQgc3RhdGUgc28gdGhlIG90aGVyIHR3byBub2Rlcydcm9uCiAgIHJ1bnMgc2VlIGl0J3MgYWxyZWFkeSBkb25lLiBObyBzaW5nbGUgbm9kZSBpcyBhIHJvdGF0aW9uIFNQT0YuCjMuICoqU2l6ZS10cmlnZ2VyZWQqKiAoYHNpemUgMjUwTWApIHJhdGhlciB0aGFuIGNhbGVuZGFyLXRyaWdnZXJlZCAoYGRhaWx5YCkgLS0KICAgdGhpcyBmaWxlIGNhbiBncm93IGZhc3QgdW5kZXIgYnVyc3RzIChzZWUgaW5jaWRlbnQgYmFja2dyb3VuZCBhYm92ZSk7IGEKICAgcHVyZWx5IGRhaWx5IGludGVydmFsIHdvdWxkIHN0aWxsIGxldCBpdCBiYWxsb29uIGJldHdlZW4gcnVucy4gQ3JvbiBjaGVja3MKICAgZXZlcnkgMTUgbWludXRlcywgc28gaXQgY2Fubm90IGdyb3cgbXVjaCBwYXN0IHRoZSAyNTBNIHRocmVzaG9sZCBpbiBwcmFjdGljZS4KCiMjIEluc3RhbGwgKG9uZS10aW1lLCBwZXIgbm9kZSkKCk11c3QgcnVuIG9uICoqYWxsIHRocmVlKiogZG9ja2VyIExYQ3MgLS0gdGhpcyBpcyBob3N0LWxldmVsIGNyb24vbG9ncm90YXRlCmNvbmZpZywgbm90IHNvbWV0aGluZyBgc3RhY2stZGVwbG95LnNoYCBjYW4gcmVhY2ggKGl0IG9ubHkgdG91Y2hlcyBTd2FybQpzZXJ2aWNlcywgbm90IGhvc3QgY3JvbiBqb2JzKS4KCmBgYGJhc2gKIyBPbiBlYWNoIG9mIGRvY2tlci0xICgxOTIuMTY4LjQuMzEpLCBkb2NrZXItMiAoMTkyLjE2OC40LjMyKSwgZG9ja2VyLTMgKDE5Mi4xNjguNC4zMyk6CnNzaCByb290QDwxOTIuMTY4LjQuMzF8LjMyfC4zMz4KY2QgL3ZvbHVtZTEvZG9ja2VyL2NvbXBvc2UtZmlsZXMKYmFzaCBkZXBsb3kvZ2l0LWd1YXJkLnNoICAgICAgICAgIyBjb25maXJtIHN5bmMgZmlyc3QsIGFzIGFsd2F5cwpzdWRvIGJhc2ggdHJhZWZpay9sb2dyb3RhdGUvaW5zdGFsbC5zaAogYGBgCgpWZXJpZnkgb24gZWFjaCBub2RlOgpgYGBiYXNoCmNhdCAvZXRjL2Nyb24uZC90cmFlZmlrLWxvZ3JvdGF0ZQpsb2dyb3RhdGUgLWQgL2V0Yy9sb2dyb3RhdGUuZC90cmFlZmlrLWxvZ3MgICAjIGRyeS1ydW4sIGNvbmZpcm1zIHN5bnRheAogYGBgCgojIyBCb290c3RyYXBwaW5nIC0tIHNocmlua2luZyB0aGUgKmVYaXN0aW5nKiBvdmVyc2l6ZWQgbG9nIGZpbGUKCioqTm90IGRvbmUgYXV0b21hdGljYWxseSBieSBgaW5zdGFsbC5zaGAuKiogVGhlIG5ldyBzaXplLXRyaWdnZXJlZCBjb25maWcgb25seQpwcmV2ZW50cyAqZnV0dXJlKiB1bmJvdW5kZWQgZ3Jvd3RoIC0tIGl0IHdvbid0IHRvdWNoIHRoZSBjdXJyZW50IDE1LjVHQiBmaWxlCnVudGlsIHRoZSBuZXh0IHRpbWUgaXQgY3Jvc3NlcyAyNTBNIChpLmUuIG5ldmVyLCBzaW5jZSBpdCdzIGFscmVhZHkgd2VsbCBwYXN0CnRoYXQgYW5kIGxvZ3JvdGF0ZSBvbmx5IGFjdHMgb24gY3Jvc3NpbmcgdGhlIHRocmVzaG9sZCBnb2luZyBmb3J3YXJkIGZyb20gaXRzCnJlY29yZGVkIHNpemUgYXQgbGFzdCBjaGVjaykuCgpgY29weXRydW5jYXRlYCBhbHdheXMgY29waWVzIHRoZSBmdWxsIGN1cnJlbnQgZmlsZSBiZWZvcmUgdHJ1bmNhdGluZyBpdCAtLSB0aGF0J3MKaW5oZXJlbnQgdG8gaG93IGl0IHdvcmtzLCBub3QgYSBidWcuIEZvcmNpbmcgYSByb3RhdGlvbiBvZiB0aGUgY3VycmVudCAxNS41R0IKZmlsZSB3b3VsZCBtb21lbnRhcmlseSBuZWVkIHJvdWdobHkgYW5vdGhlciAxNS1HQi1zaXplZCBjaHVuayBvZiBmcmVlIHNwYWNlIC0tCmFuZCBgL3ZvbHVtZTEvZG9ja2VyLXJvb3RgIG9ubHkgaGFkICoqfjE0RyBmcmVlKiogYXQgbGFzdCBjaGVjay4gRG9pbmcgdGhpcwpibGluZGx5IGNvdWxkIHRpcCB0aGUgdm9sdW1lIHRvIDEwMCUgbWlkLW9wZXJhdGlvbi4KCioqUmVjb21tZW5kZWQgbWFudWFsIHN0ZXAgKG9wZXJhdG9yLXJ1biwgb24gYW55IG9uZSBub2RlIC0tIGl0J3MgdGhlIHNhbWUKQ2VwaEZTIGZpbGUgZnJvbSBhbGwgdGhyZWUpKio6IHRoZXNlIGFyZSBkaWFnbm9zdGljIGFjY2Vzcy9kZWJ1ZyBsb2dzLCBub3Qgc29tZXRoaW5nCndvcnRoIHByZXNlcnZpbmcgaW4gZnVsbCwgc28ganVzdCB0cnVuY2F0ZSBkaXJlY3RseSByYXRoZXIgdGhhbiBjb21wcmVzcy10aGVuLQp0cnVuY2F0ZTogCgpgYGBiYXNoCiMgT3B0aW9uYWw6IGtlZXAgYSBzbWFsbCB0YWlsIHNhbXBsZSBmb3IgcmVmZXJlbmNlIGJlZm9yZSB0cnVuY2F0aW5nCnRhaWwgLWMgNTAwMDAwMDAgL3ZvbHVtZTEvZG9ja2VyL3RyYWVmaWsvbG9ncy9hY2Nlc3MubG9nID4gXAogIC92b2x1bWUxL2RvY2tlci90cmFlZmlrLWFjY2Vzcy1sb2ctYXJjaGl2ZS0kKGRhdGUgKyVZJW0lZCkubG9nCgojIFRoZW4gdHJ1bmNhdGUgaW4gcGxhY2UgKHNhZmUgLS0gVHJhZWZpaydzIGV4aXN0aW5nIGZpbGUgaGFuZGxlcyBvbiBhbGwgMwojIG5vZGVzIHN0YXkgdmFsaWQsIHNhbWUgYXMgY29weXRydW5jYXRlJ3Mgb3duIG1lY2hhbmlzbSk6CjogPiAvdm9sdW1lMS9kb2NrZXIvdHJhZWZpay9sb2dzL2FjY2Vzcy5sb2cKOiA+IC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MvdHJhZWZpay5sb2cKCiMgQ29uZmlybToKZGYgLWggL3ZvbHVtZTEvZG9ja2VyLXJvb3QKbHMgLWxhIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MvCmBgYAoKQWZ0ZXIgdGhpcyBvbmUtdGltZSBib290c3RyYXAsIHRoZSBjcm9uK2xvZ3JvdGF0ZSBzZXR1cCBrZWVwcyBpdCBib3VuZGVkCihyb3RhdGVzIGF0IDI1ME0sIGtlZXBzIDQ4IGNvbXByZXNzZWQgZ2VuZXJhdGlvbnMsIHBydW5lcyBhbnl0aGluZyBvdmVyIDE0CmRheXMgb2xkKSBnb2luZyBmb3J3YXJkIHdpdGhvdXQgbmVlZGluZyBhbnkgZnVydGhlciBtYW51YWwgaW50ZXJ2ZW50aW9uLgoKIyMgUmVsYXRlZCwgTk9UIGluY2x1ZGVkIGluIHRoaXMgY2hhbmdlIChmb2xsb3ctdXAgdG8gY29uc2lkZXIgc2VwYXJhdGVseSkKCmB0cmFlZmlrLnlhbWxgIGN1cnJlbnRseSBydW5zIGAtLWxvZy5sZXZlbD1EQlVHYCAtLSB2ZXJib3NlIGRlYnVnIGxvZ2dpbmcgaW4KcHJvZHVjdGlvbiwgd2hpY2ggaXMgYSBtZWFuaW5nZnVsIGNvbnRyaWJ1dG9yIHRvIGhvdyBmYXN0IHRoZXNlIGZpbGVzIGdyb3cuCkxvd2VyaW5nIHRvIGBJTkZPYCB3b3VsZCByZWR1Y2Ugdm9sdW1lIHNpZ25pZmljYW50bHkgYnV0IHJlcXVpcmVzIGEgcmVhbAptb2RpZmljYXRpb24gK3JlZGVwbG95IG9mIHRoZSBoaWdoLWJsYXN0LXJhZGl1cyBgdHJhZWZpa2Agc3RhY2sgKGFsbCBIVFRQL0hUVFBTCApyb3V0aW5nIGRlcGVuZHMgb24gaXQpLCBzbyBpdCdzIGludGVudGlvbmFsbHkgbGVmdCBvdXQgb2YgdGhpcyBQUiBhbmQgc2hvdWxkCmJlIGl0cyBvd24gcmV2aWV3ZWQgY2hhbmdlIGlmIHdhbnRlZC4= \ No newline at end of file From 8ef1cea5b0d8488e38286e26c89406644a742021 Mon Sep 17 00:00:00 2001 From: Bot Date: Fri, 11 Sep 2026 23:29:55 -0700 Subject: [PATCH 5/9] traefik: fix malformed LOCK_FILE variable in rotation wrapper --- traefik/logrotate/traefik-logrotate.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/traefik/logrotate/traefik-logrotate.sh b/traefik/logrotate/traefik-logrotate.sh index c27a8dc..1242043 100644 --- a/traefik/logrotate/traefik-logrotate.sh +++ b/traefik/logrotate/traefik-logrotate.sh @@ -1 +1 @@ -IyEvdXNyL2Jpbi9lbnYgYmFzaAojIFdyYXBwZXIgaW52b2tlZCBieSBjcm9uIG9uIGRvY2tlci0xL2RvY2tlci0yL2RvY2tlci0zIHRvIHJvdGF0ZSB0aGUgc2hhcmVkCiMgVHJhZWZpayBhY2Nlc3MvZXJyb3IgbG9ncyAoc2VlIHRyYWVmaWstbG9ncy5jb25mIGZvciB0aGUgImZ1bGwgd2h5IikuCiMKIyBCZWNhdXNlIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MgaXMgdGhlIFNBTUUgcGh5c2ljYWwgQ2VwaEZTIHBhdGggb24KIyBhbGwgdGhyZWUgbm9kZXMsIGFuZCBjcm9uIG9uIGFsbCB0aHJlZSBub2RlcyBydW5zIHRoaXMgaW5kZXBlbmRlbnRseSwgd2UKIyB1c2UgYSBzaGFyZWQgZmxvY2sgKGFsc28gb24gdGhlIENlcGhGUyBtb3VudCwgc28gaXQncyB2aXNpYmxlIGNsdXN0ZXItd2lkZSkKIyB0byBndWFyYW50ZWUgb25seSBvbmUgbm9kZSBhY3R1YWxseSBleGVjdXRlcyBsb2dyb3RhdGUgYXQgYSB0aW1lLCBhbmQgYQojIFNIQVJFRCBzdGF0ZSBmaWxlIHNvIHdoaWNoZXZlciBub2RlIHJ1bnMgaXQga25vd3MgdGhlIHRydWUgbGFzdC1yb3RhdGVkCiMgdGltZSByZWdhcmRsZXNzIG9mIHdoaWNoIG5vZGUgcm90YXRlZCBpdCBsYXN0LiBJZiBhIG5vZGUgaXMgZG93biwgdGhlCiMgb3RoZXIgdHdvIHN0aWxsIGNvdmVyIHRoZSBzY2hlZHVsZSAtLSBub25lIG9mIHRoaXMgcmVsaWVzIG9uIGEgc3BlY2lmaWMKIyBub2RlIGJlaW5nIHVwLgpzZXQgLWV1byBwaXBlZmFpbAoKTE9DS19ESVI9Ii92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3JvdGF0ZS1zdGF0ZSIKTE9DS19GSUxFPSIkeyBMT0NLX0RJUn0vcm90YXRlLmxvY2siCkxPQ0tfRklMRT0iJHtMT0NLX0RJUn0vcm90YXRlLmxvY2siClNUQVRFX0ZJTEU9IiR7TE9DS19ESVJ9L3N0YXR1cyIKQ09ORl9GSUxFPSIvZXRjL2xvZ3JvdGF0ZS5kL3RyYWVmaWstbG9ncyIKCm1rZGlyIC1wICIkTE9DS19ESVIiCnRvdWNoICIkU1RBVEVfRklMRSIKCmV4ZWMgMjAwPiIkTE9DS19GSUxFIgppZiAhIGZsb2NrIC1uIDIwMDsgdGhlbgogICMgQW5vdGhlciBub2RlIGFscmVhZHkgaG9sZHMgdGhlIGxvY2sgdGhpcyBjeWNsZSAtLSBub3JtYWwsIG5vdCBhbiBlcnJvci4KICBleGl0IDAKZmkKCi91c3Ivc2Jpbi9sb2dyb3RhdGUgLXMgIiRTVEFURV9GSUxFIiAiJENPTkZfRklMRSI= \ No newline at end of file +IyEvdXNyL2Jpbi9lbnYgYmFzaAojIFdyYXBwZXIgaW52b2tlZCBieSBjcm9uIG9uIGRvY2tlci0xL2RvY2tlci0yL2RvY2tlci0zIHRvIHJvdGF0ZSB0aGUgc2hhcmVkCiMgVHJhZWZpayBhY2Nlc3MvZXJyb3IgbG9ncyAoc2VlIHRyYWVmaWstbG9ncy5jb25mIGZvciB0aGUgImZ1bGwgd2h5IikuCiMKIyBCZWNhdXNlIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MgaXMgdGhlIFNBTUUgcGh5c2ljYWwgQ2VwaEZTIHBhdGggb24KIyBhbGwgdGhyZWUgbm9kZXMsIGFuZCBjcm9uIG9uIGFsbCB0aHJlZSBub2RlcyBydW5zIHRoaXMgaW5kZXBlbmRlbnRseSwgd2UKIyB1c2UgYSBzaGFyZWQgZmxvY2sgKGFsc28gb24gdGhlIENlcGhGUyBtb3VudCwgc28gaXQncyB2aXNpYmxlIGNsdXN0ZXItd2lkZSkKIyB0byBndWFyYW50ZWUgb25seSBvbmUgbm9kZSBhY3R1YWxseSBleGVjdXRlcyBsb2dyb3RhdGUgYXQgYSB0aW1lLCBhbmQgYQojIFNIQVJFRCBzdGF0ZSBmaWxlIHNvIHdoaWNoZXZlciBub2RlIHJ1bnMgaXQga25vd3MgdGhlIHRydWUgbGFzdC1yb3RhdGVkCiMgdGltZSByZWdhcmRsZXNzIG9mIHdoaWNoIG5vZGUgcm90YXRlZCBpdCBsYXN0LiBJZiBhIG5vZGUgaXMgZG93biwgdGhlCiMgb3RoZXIgdHdvIHN0aWxsIGNvdmVyIHRoZSBzY2hlZHVsZSAtLSBub25lIG9mIHRoaXMgcmVsaWVzIG9uIGEgc3BlY2lmaWMKIyBub2RlIGJlaW5nIHVwLgpzZXQgLWV1byBwaXBlZmFpbAoKTE9DS19ESVI9Ii92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3JvdGF0ZS1zdGF0ZSIKTE9DS19GSUxFPSIkTE9DS19ESVIvcm90YXRlLmxvY2siClNUQVRFX0ZJTEU9IiRMT0NLX0RJUi9zdGF0dXMiCkNPTkZfRklMRT0iL2V0Yy9sb2dyb3RhdGUuZC90 \ No newline at end of file From a2df13569394034cdd1b61c39cd2b3c306ac5563 Mon Sep 17 00:00:00 2001 From: Bot Date: Sat, 12 Sep 2026 00:20:56 -0700 Subject: [PATCH 6/9] traefik: fix corrupted logrotate config (was double base64-encoded) --- traefik/logrotate/traefik-logs.conf | 44 ++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/traefik/logrotate/traefik-logs.conf b/traefik/logrotate/traefik-logs.conf index ac6add8..feb80e5 100644 --- a/traefik/logrotate/traefik-logs.conf +++ b/traefik/logrotate/traefik-logs.conf @@ -1 +1,43 @@ -IyBUcmFlZmlrIGFjY2Vzcy9lcnJvciBsb2cgcm90YXRpb24KIwojIENPTlRFWFQ6IHRyYWVmaWtfcmV2ZXJzZS1wcm94eSBydW5zIGluIFN3YXJtIGBtb2RlOiBnbG9iYWxgICh0cmFlZmlrLnlhbWwpLAojIG1lYW5pbmcgb25lIGluc3RhbmNlIHJ1bnMgb24gRUFDSCBvZiBkb2NrZXItMS9kb2NrZXItMi9kb2NrZXItMy4gQWxsIHRocmVlCiMgd3JpdGUgdG8gdGhlIFNBTUUgcGh5c2ljYWwgZmlsZSB2aWEgdGhlIHNoYXJlZCBDZXBoRlMgYmluZCBtb3VudAojIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MgLT4gL3RyYWVmaWsvbG9ncyAoaWRlbnRpY2FsIHBhdGggZnJvbSBhbnkgbm9kZSAtLQojIHNlZSBpbmZyYSBjb250ZXh0OiAvdm9sdW1lMS9kb2NrZXIgaXMgYSBzaGFyZWQgQ2VwaEZTIG1vdW50KS4KIwojIGBjb3B5dHJ1bmNhdGVgIGlzIFJFUVVJUkVEIGhlcmUgKG5vdCB0aGUgZGVmYXVsdCBjcmVhdGUrc2lnbmFsIGFwcHJvYWNoKToKIyBUcmFlZmlrIGhhcyBubyBTSUdVU1IxL1NJR0hVUCAicmVvcGVuIGxvZyBmaWxlIiBoYW5kbGluZywgYW5kIGV2ZW4gaWYgaXQKIyBkaWQsIGNvb3JkaW5hdGluZyBhIHJlb3BlbiBzaWduYWwgYWNyb3NzIDMgaW5kZXBlbmRlbnQgcGVyLW5vZGUgY29udGFpbmVycwojIHdyaXRpbmcgdG8gb25lIHNoYXJlZCBpbm9kZSBpcyB1bm5lY2Vzc2FyeSBjb21wbGV4aXR5LiBjb3B5dHJ1bmNhdGUga2VlcHMKIyBldmVyeSB3cml0ZXIncyBleGlzdGluZyBmaWxlIGRlc2NyaXB0b3IgdmFsaWQgKHRydW5jYXRlcyBpbiBwbGFjZSkgc28gYWxsCiMgdGhyZWUgVHJhZWZpayBwcm9jZXNzZXMga2VlcCBhcHBlbmRpbmcgdG8gdGhlIHNhbWUgaW5vZGUgd2l0aCB6ZXJvCiMgc2lnbmFsaW5nLiBUcmFkZW9mZjogYSBoYW5kZnVsIG9mIGxvZyBsaW5lcyB3cml0dGVuIGluIHRoZSBleGFjdAojIGNvcHkvdHJ1bmNhdGUgaW5zdGFudCBjYW4gYmUgbG9zdCAtLSBhY2NlcHRhYmxlIGZvciBkaWFnbm9zdGljIGFjY2Vzcy9lcnJvcgojIGxvZ3MsIG5vdCB1c2VkIGZvciBhbnl0aGluZyB0cmFuc2FjdGlvbmFsLgojCiMgU2l6ZS10cmlnZ2VyZWQgKG5vdCBjYWxlbmRhci10cmlnZ2VyZWQpIG9uIHB1cnBvc2U6IHRoaXMgZmlsZSBjYW4gZ3JvdyBmYXN0CiMgdW5kZXIgYnVyc3RzIChzZWUgaW5jaWRlbnQgdGhhdCBwcm9tcHRlZCB0aGlzIC0tIDE1LjVHQiBhY2N1bXVsYXRlZCB3aXRoCiMgLS1sb2cubGV2ZWw9REVCVUcgc2V0KS4gYHNpemVgIGlzIGNoZWNrZWQgZXZlcnkgdGltZSB0aGUgd3JhcHBlciBzY3JpcHQgcnVucwojIChjcm9uLCBldmVyeSAxNSBtaW51dGVzIC0tIHNlZSBpbnN0YWxsLnNoKSwgc28gaXQgY2Fubm90IGJhbGxvb24gdW5ib3VuZGVkCiMgYmV0d2VlbiBjaGVja3MgdGhlIHdheSBhIHBsYWluIGBkYWlseWAgaW50ZXJ2YWwgd291bGQuCiMKIyBJbnN0YWxsZWQgdmlhIGluc3RhbGwuc2ggb24gQUxMIFRIUkVFIGRvY2tlciBMWENzIChkb2NrZXItMSwgZG9ja2VyLTIsCiMgZG9ja2VyLTMpIC0tIHNlZSBSRUFETUUubWQuIFRoaXMgaXMgYSBIT1NULWxldmVsIGNyb24vbG9ncm90YXRlIGNvbmZpZywKIyBvdXRzaWRlIHRoZSBXb29kcGVja2VyL3N0YWNrLWRlcGxveS5zaCBwaXBlbGluZSAoZG9ja2VyIHN0YWNrIGRlcGxveSBoYXMgbm8KIyBtZWNoYW5pc20gdG8gdG91Y2ggaG9zdCBjcm9uKSwgc28gaXQgbXVzdCBiZSBhcHBsaWVkIG1hbnVhbGx5IG9uY2UgcGVyIG5vZGUsCiMgbm90IHZpYSBhIHN0YWNrIHJlZGVwbG95LgoKL3ZvbHVtZTEvZG9ja2VyL3RyYWVmaWsvbG9ncy8qLmxvZyB7CiAgICBzaXplIDI1ME0KICAgIHJvdGF0ZSA0OAogICAgbWF4YWdlIDE0CiAgICBjb21wcmVzcwogICAgZGVsYXljb21wcmVzcwogICAgbWlzc2luZ29rCiAgICBub3RpZmVtcHR5CiAgICBjb3B5dHJ1bmNhdGUKICAgIGRhdGVleHQKICAgIGRhdGVmb3JtYXQgLSVZJW0lZC0lSCVNJVMKICAgIHN1IHJvb3Qgcm9vdAp9Cg== \ No newline at end of file +# Traefik access/error log rotation +# +# CONTEXT: traefik_reverse-proxy runs in Swarm `mode: global` (traefik.yaml), +# meaning one instance runs on EACH of docker-1/docker-2/docker-3. All three +# write to the SAME physical file via the shared CephFS bind mount +# /volume1/docker/traefik/logs -> /traefik (identical path from any node -- +# see infra context: /volume1/docker is a shared CephFS mount). +# +# `copytruncate` is REQUIRED here (not the default create+signal approach): +# Traefik has no SIGUSR1/SIGHUP "reopen log file" handling, and even if it +# did, coordinating a reopen signal across 3 independent per-node containers +# writing to one shared inode is unnecessary complexity. copytruncate keeps +# every writer's existing file descriptor valid (truncates in place) so all +# three Traefik processes keep appending to the same inode with zero +# signaling. Tradeoff: a handful of log lines written in the exact +# copy/truncate instant can be lost -- acceptable for diagnostic access/error +# logs, not used for anything transactional. +# +# Size-triggered (not calendar-triggered) on purpose: this file can grow fast +# under bursts (see incident that prompted this -- 15.5GB accumulated with +# --log.level=DEBUG set). `size` is checked every time the wrapper script runs +# (cron, every 15 minutes -- see install.sh), so it cannot balloon unbounded +# between checks the way a plain `daily` interval would. +# +# Installed via install.sh on ALL THREE docker LXCs (docker-1, docker-2, +# docker-3) -- see README.md. This is a HOST-level cron/logrotate config, +# outside the Woodpecker/stack-deploy.sh pipeline (docker stack deploy has no +# mechanism to touch host cron), so it must be applied manually once per node, +# not via a stack redeploy. + +/volume1/docker/traefik/logs/*.log { + size 250M + rotate 48 + maxage 14 + compress + delaycompress + missingok + notifempty + copytruncate + dateext + dateformat -%Y%m%d-%H%M%S + su root root +} From 8c2ee593d49b5b7f6f57319d016de5ac3c3457a6 Mon Sep 17 00:00:00 2001 From: Bot Date: Sat, 12 Sep 2026 00:20:56 -0700 Subject: [PATCH 7/9] traefik: fix corrupted logrotate wrapper script (was double base64-encoded) --- traefik/logrotate/traefik-logrotate.sh | 30 +++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/traefik/logrotate/traefik-logrotate.sh b/traefik/logrotate/traefik-logrotate.sh index 1242043..5ddfe44 100644 --- a/traefik/logrotate/traefik-logrotate.sh +++ b/traefik/logrotate/traefik-logrotate.sh @@ -1 +1,29 @@ -IyEvdXNyL2Jpbi9lbnYgYmFzaAojIFdyYXBwZXIgaW52b2tlZCBieSBjcm9uIG9uIGRvY2tlci0xL2RvY2tlci0yL2RvY2tlci0zIHRvIHJvdGF0ZSB0aGUgc2hhcmVkCiMgVHJhZWZpayBhY2Nlc3MvZXJyb3IgbG9ncyAoc2VlIHRyYWVmaWstbG9ncy5jb25mIGZvciB0aGUgImZ1bGwgd2h5IikuCiMKIyBCZWNhdXNlIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MgaXMgdGhlIFNBTUUgcGh5c2ljYWwgQ2VwaEZTIHBhdGggb24KIyBhbGwgdGhyZWUgbm9kZXMsIGFuZCBjcm9uIG9uIGFsbCB0aHJlZSBub2RlcyBydW5zIHRoaXMgaW5kZXBlbmRlbnRseSwgd2UKIyB1c2UgYSBzaGFyZWQgZmxvY2sgKGFsc28gb24gdGhlIENlcGhGUyBtb3VudCwgc28gaXQncyB2aXNpYmxlIGNsdXN0ZXItd2lkZSkKIyB0byBndWFyYW50ZWUgb25seSBvbmUgbm9kZSBhY3R1YWxseSBleGVjdXRlcyBsb2dyb3RhdGUgYXQgYSB0aW1lLCBhbmQgYQojIFNIQVJFRCBzdGF0ZSBmaWxlIHNvIHdoaWNoZXZlciBub2RlIHJ1bnMgaXQga25vd3MgdGhlIHRydWUgbGFzdC1yb3RhdGVkCiMgdGltZSByZWdhcmRsZXNzIG9mIHdoaWNoIG5vZGUgcm90YXRlZCBpdCBsYXN0LiBJZiBhIG5vZGUgaXMgZG93biwgdGhlCiMgb3RoZXIgdHdvIHN0aWxsIGNvdmVyIHRoZSBzY2hlZHVsZSAtLSBub25lIG9mIHRoaXMgcmVsaWVzIG9uIGEgc3BlY2lmaWMKIyBub2RlIGJlaW5nIHVwLgpzZXQgLWV1byBwaXBlZmFpbAoKTE9DS19ESVI9Ii92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3JvdGF0ZS1zdGF0ZSIKTE9DS19GSUxFPSIkTE9DS19ESVIvcm90YXRlLmxvY2siClNUQVRFX0ZJTEU9IiRMT0NLX0RJUi9zdGF0dXMiCkNPTkZfRklMRT0iL2V0Yy9sb2dyb3RhdGUuZC90 \ No newline at end of file +#!/usr/bin/env bash +# Wrapper invoked by cron on docker-1/docker-2/docker-3 to rotate the shared +# Traefik access/error logs (see traefik-logs.conf for the "full why"). +# +# Because /volume1/docker/traefik/logs is the SAME physical CephFS path on +# all three nodes, and cron on all three nodes runs this independently, we +# use a shared flock (also on the CephFS mount, so it's visible cluster-wide) +# to guarantee only one node actually executes logrotate at a time, and a +# SHARED state file so whichever node runs it knows the true last-rotated +# time regardless of which node rotated it last. If a node is down, the +# other two still cover the schedule -- none of this relies on a specific +# node being up. +set -euo pipefail + +LOCK_DIR="/volume1/docker/traefik/logrotate-state" +LOCK_FILE="$LOCK_DIR/rotate.lock" +STATE_FILE="$LOCK_DIR/status" +CONF_FILE="/etc/logrotate.d/traefik-logs" + +mkdir -p "$LOCK_DIR" +touch "$STATE_FILE" + +exec 200>"$LOCK_FILE" +if ! flock -n 200; then + # Another node already holds the lock this cycle -- normal, not an error. + exit 0 +fi + +/usr/sbin/logrotate -s "$STATE_FILE" "$CONF_FILE" From 0a334fa7819138d272cfbc40356f62b001125330 Mon Sep 17 00:00:00 2001 From: Bot Date: Sat, 12 Sep 2026 00:21:10 -0700 Subject: [PATCH 8/9] traefik: fix corrupted install.sh (was double base64-encoded) --- traefik/logrotate/install.sh | 45 +++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/traefik/logrotate/install.sh b/traefik/logrotate/install.sh index 27f0f6a..bf8d397 100644 --- a/traefik/logrotate/install.sh +++ b/traefik/logrotate/install.sh @@ -1 +1,44 @@ -IyEvdXNyL2Jpbi9lbnYgYmFzaAojIE9uZS10aW1lIGluc3RhbGxlciBmb3IgVHJhZWZpayBsb2cgcm90YXRpb24uCiMKIyBNVVNUIGJlIHJ1biBtYW51YWxseSwgb25jZSwgb24gRUFDSCBvZiBkb2NrZXItMSwgZG9ja2VyLTIsIGRvY2tlci0zLgojIFRoaXMgaXMgaW50ZW50aW9uYWxseSBOT1QgcGFydCBvZiBzdGFjay1kZXBsb3kuc2ggLyB0aGUgV29vZHBlY2tlcgojIHBpcGVsaW5lOiBpdCBpbnN0YWxscyBhIGhvc3QtbGV2ZWwgY3Jvbi5kIGVudHJ5IGFuZCAvZXRjL2xvZ3JvdGF0ZS5kCiMgY29uZmlnLCBhbmQgYGRvY2tlciBzdGFjayBkZXBsb3lgIGhhcyBubyBtZWNoYW5pc20gdG8gcmVhY2ggb3V0c2lkZSB0aGUKIyBTd2FybS9jb250YWluZXIgYm91bmRhcnkgb250byBob3N0IGNyb24uIFNlZSBSRUFETUUubWQgZm9yIHdoeSB0aGlzIG5lZWRzCiMgdG8gcnVuIG9uIGFsbCB0aHJlZSBub2Rlcy4KIwojIFVzYWdlIChmcm9tIGEgY2hlY2tvdXQgb2YgdGhpcyByZXBvLCBvbiBlYWNoIG5vZGUpOgojICAgc3VkbyBiYXNoIHRyYWVmaWsvbG9ncm90YXRlL2luc3RhbGwuc2gKc2V0IC1ldW8gcGlwZWZhaWwKCmlmIFtbICRFVUlEIC1uZSAwIF1dOyB0aGVuCiAgZWNobyAiUnVuIGFzIHJvb3QgKHN1ZG8pLiIgPiYyCiAgZXhpdCAxCmZpCgpSRVBPX0RJUj0iJChjZCAiJChkaXJuYW1lICIke0JBU0hfU09VUkNFWzBdfSIpIiAmJiBwd2QpIgoKZWNobyAiSW5zdGFsbGluZyBsb2dyb3RhdGUgY29uZmlnLi4uIgppbnN0YWxsIC1tIDA2NDQgIiRSRVBPX0RJUi90cmFlZmlrLWxvZ3MuY29uZiIgL2V0Yy9sb2dyb3RhdGUuZC90cmFlZmlrLWxvZ3MKCmVjaG8gIkluc3RhbGxpbmcgcm90YXRpb24gd3JhcHBlci4uLiIKaW5zdGFsbCAtbSAwNzU1ICIkUkVQT19ESVIvdHJhZWZpay1sb2dyb3RhdGUuc2giIC91c3IvbG9jYWwvc2Jpbi90cmFlZmlrLWxvZ3JvdGF0ZS5zaAoKZWNobyAiSW5zdGFsbGluZyBjcm9uLmQgc2NoZWR1bGUgKGV2ZXJ5IDE1IG1pbnV0ZXMpLi4uIgpjYXQgPiAvZXRjL2Nyb24uZC90cmFlZmlrLWxvZ3JvdGF0ZSA8PCAnRU9GJwojIE1hbmFnZWQgYnkgaG9tZWxhYi9jb21wb3NlLWZpbGVzIHRyYWVmaWsvbG9ncm90YXRlL2luc3RhbGwuc2ggLS0gZG8gbm90CiMgaGFuZC1lZGl0OyB1cGRhdGUgdHJhZWZpay9sb2dyb3RhdGUvKi57Y29uZixzaH0gaW4gR2l0ZWEgYW5kIHJlLXJ1bgojIGluc3RhbGwuc2ggaW5zdGVhZC4KKi8xNSAqICogKiAqIHJvb3QgL3Vzci9sb2NhbC9zYmluL3RyYWVmaWstbG9ncm90YXRlLnNoCkVPRgpjaG1vZCAwNjQ0IC9ldGMvY3Jvbi5kL3RyYWVmaWstbG9ncm90YXRlCgplY2hvICJWZXJpZnlpbmcgbG9ncm90YXRlIGNvbmZpZyBzeW50YXguLi4iCmxvZ3JvdGF0ZSAtZCAvZXRjL2xvZ3JvdGF0ZS5kL3RyYWVmaWstbG9ncwoKZWNobyAiRG9uZS4gRmlyc3Qgcm90YXRpb24gY2hlY2sgcnVucyBvbiB0aGUgbmV4dCBjcm9uIHRpY2sgKHVwIHRvIDE1IG1pbikuIgplY2hvICJUaGlzIG9ubHkgcm90YXRlcyBnb2luZyBmb3J3YXJkIG9uY2UgdGhlIGZpbGUgY3Jvc3NlcyB0aGUgc2l6ZSB0aHJlc2hvbGQuIgplY2hvICJUbyBzaHJpbmsgdGhlIEVYSVNUSU5HIGFscmVhZHktbGFyZ2UgbG9nIGZpbGUsIHNlZSBSRUFETUUubWQgLS0gdGhhdCBpcyIKZWNobyAiYSBzZXBhcmF0ZSwgZGVsaWJlcmF0ZSBtYW51YWwgc3RlcCAobm90IGRvbmUgYnkgdGhpcyBzY3JpcHQpIGJlY2F1c2UgaXQiCmVjaG8gIm5lZWRzIGRpc2sgaGVhZHJvb20gYXdhcmVuZXNzIGZpcnN0LiI= \ No newline at end of file +#!/usr/bin/env bash +# One-time installer for Traefik log rotation. +# +# MUST be run manually, once, on EACH of docker-1, docker-2, docker-3. +# This is intentionally NOT part of stack-deploy.sh / the Woodpecker +# pipeline: it installs a host-level cron.d entry and /etc/logrotate.d +# config, and `docker stack deploy` has no mechanism to reach outside the +# Swarm/container boundary onto host cron. See README.md for why this needs +# to run on all three nodes. +# +# Usage (from a checkout of this repo, on each node): +# sudo bash traefik/logrotate/install.sh +set -euo pipefail + +if [[ $EUID -ne 0 ]]; then + echo "Run as root (sudo)." >&2 + exit 1 +fi + +REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +echo "Installing logrotate config..." +install -m 0644 "$REPO_DIR/traefik-logs.conf" /etc/logrotate.d/traefik-logs + +echo "Installing rotation wrapper..." +install -m 0755 "$REPO_DIR/traefik-logrotate.sh" /usr/local/sbin/traefik-logrotate.sh + +echo "Installing cron.d schedule (every 15 minutes)..." +cat > /etc/cron.d/traefik-logrotate << 'EOF' +# Managed by homelab/compose-files traefik/logrotate/install.sh -- do not +# hand-edit; update traefik/logrotate/*.{conf,sh} in Gitea and re-run +# install.sh instead. +*/15 * * * * root /usr/local/sbin/traefik-logrotate.sh +EOF +chmod 0644 /etc/cron.d/traefik-logrotate + +echo "Verifying logrotate config syntax..." +logrotate -d /etc/logrotate.d/traefik-logs + +echo "Done. First rotation check runs on the next cron tick (up to 15 min)." +echo "This only rotates going forward once the file crosses the size threshold." +echo "To shrink the EXISTING already-large log file, see README.md -- that is" +echo "a separate, deliberate manual step (not done by this script) because it" +echo "needs disk headroom awareness first." From a2308b76f80240971e29d76af1dc380622d47fbe Mon Sep 17 00:00:00 2001 From: Bot Date: Sat, 12 Sep 2026 00:21:36 -0700 Subject: [PATCH 9/9] traefik: fix corrupted README (was double base64-encoded) --- traefik/logrotate/README.md | 121 +++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/traefik/logrotate/README.md b/traefik/logrotate/README.md index d8c3b75..ce5055b 100644 --- a/traefik/logrotate/README.md +++ b/traefik/logrotate/README.md @@ -1 +1,120 @@ -IyBUcmFlZmlrIExvZyBSb3RhdGlvbgoKIyMgV2h5IHRoaXMgZXhpc3RzCgpgdHJhZWZpay95YW1sYCBydW5zIFRyYWVmaWsgd2l0aDoKLSBgLS1hY2Nlc3Nsb2cuZmlsZVBhdGg9L3RyYWVmaWsvbG9ncy9hY2Nlc3MubG9nYAotIGAtLWxvZy5maWxlUGF0aD0vdHJhZWZpay9sb2dzL3RyYWVmaWsubG9nYAotIGAtLWxvZy5sZXZlbD1EQlVHYCAoc2ljIC0gYWN0dWFsbHkgYERFQlVHYCkKCk5laXRoZXIgZmlsZSBoYXMgYW55IGJ1aWx0LWluIHJvdGF0aW9uIC0tIFRyYWVmaWsgaGFzIG5vIG5hdGl2ZSByb3RhdGUtb24tc2l6ZQotIG5vciBhIFNJR1VTUjEvcmVvcGVuIGhhbmRsZXIuIERvY2tlcidzIGBqc29uLWZpbGVgIGxvZy1kcml2ZXIgcm90YXRpb24KKGBtYXgtc2l6ZWAvYG1heC1maWxlYCkgb25seSBhcHBsaWVzIHRvIHN0ZG91dCwgbm90IHRvIGZpbGVzIFRyYWVmaWsgd3JpdGVzCmRpcmVjdGx5IHZpYSBgLS1hY2Nlc3Nsb2cuZmlsZVBhdGhgL2AtLWxvZy5maWxlUGF0aGAuIFJlc3VsdDogYGFjY2Vzcy5sb2dgIGdyZXcKdG8gKip+MTUuNUdCIHVucm90YXRlZCoqIGJlZm9yZSB0aGlzIHdhcyBjYXVnaHQsIG9uIGEgQ2VwaEZTIHZvbHVtZQooYC92b2x1bWUxL2RvY2tlci1yb290YCkgYWxyZWFkeSBhdCA4Mi04NCUgdXNlZC5Cb3RoIHRoZSBkaXNrIHByZXNzdXJlIGFuZAp0aGUgb25nb2luZyB3cml0ZSBsYXRlbmN5IG9mIGFwcGVuZGluZyB0byBhIDE1R0IgZmlsZSBvbiBhIG5ldHdvcmsgZmlsZXN5c3RlbQpvbiBFVkVSWSByZXF1ZXN0IHRocm91Z2ggVHJhZWZpayB3ZXJlIGZsYWdnZWQgYXMgYSByZWFsIHJpc2sgZmFjdG9yIGR1cmluZwp0cm91Ymxlc2hvb3RpbmcgKFVwdGltZSBLdW1hIFdlYlNvY2tldCBmbGFwcGluZyBpbnZlc3RpZ2F0aW9uLCBTZXAgMjAyNikuCgojIyBBcmNoaXRlY3R1cmUgY29uc3RyYWludCB0aGlzIGRlc2lnbiBhY2NvdW50cyBmb3IKCmB0cmFlZmlrX3JldmVyc2UtcHJveHlgIHJ1bnMgU3dhcm0gKmBtb2RlOiBnbG9iYWxgKiAtLSBvbmUgaW5zdGFuY2Ugb24gKiplYWNoKioKb2YgZG9ja2VyLTEsIGRvY2tlci0yLCBkb2NrZXItMy4gQWxsIHRocmVlIHdyaXRlIHRvIHRoZSAqKnNhbWUgcGh5c2ljYWwgZmlsZSoqCnZpYSB0aGUgc2hhcmVkIENlcGhGUyBiaW5kIG1vdW50IGAvdm9sdW1lMS9kb2NrZXIvdHJhZWZpayAtPiAvdHJhZWZpa2AgKGlkZW50aWNhbAptb3VudCwgdmlzaWJsZSBpZGVudGljYWxseSBmcm9tIGFueSBub2RlKS4gVGhhdCBydWxlcyBvdXQ6CgotICoqU2lnbmFsLWJhc2VkIHJvdGF0aW9uKiogKGNsYXNzaWMgYGNyZWF0ZWAgKyBgcG9zdHJvdGF0ZWAgc2VuZGluZyBTSUdVU1IxKToKICBUcmFlZmlrIGRvZXNuJ3QgaW1wbGVtZW50IGEgcmVvcGVuIHNpZ25hbCwgYW5kIGV2ZW4gaWYgaXQgZGlkLCB5b3UnZCBuZWVkIHRvCiAgc2lnbmFsIDMgc2VwYXJhdGUgcGVyLW5vZGUgY29udGFpbmVycyBpbiBsb2Nrc3RlcC4KLSAqKlJ1bm5pbmcgbG9ncm90YXRlIG9uIGp1c3Qgb25lIG5vZGUqKjogd29ya3MgdW50aWwgdGhhdCBub2RlIGlzIGRvd24sIHRoZW4KICByb3RhdGlvbiBzaWxlbnRseSBzdG9wcyB3aXRoIG5vIGFsZXJ0LgoKU28gdGhpcyBzZXR1cCB1c2VzOgoxLiAqKmBjb3B5dHJ1bmNhdGVgKiogKHNlZSBgdHJhZWZpay1sb2dzLmNvbmZgKSAtLSBhbGwgdGhyZWUgVHJhZWZpayBwcm9jZXNzZXMKICAga2VlcCB3cml0aW5nIHRvIHRoZSBzYW1lIGlub2RlLCBubyBzaWduYWxpbmcgbmVlZGVkLiBUcmFkZW9mZjogYSBmZXcgbG9nCiAgIGxpbmVzIHdyaXR0ZW4gaW4gdGhlIGV4YWN0IGNvcHkvdHJ1bmNhdGUgaW5zdGFudCBjYW4gYmUgbG9zdCAtLSBmaW5lIGZvcgogICBkaWFnbm9zdGljIGxvZ3MuCjIuICoqQ3JvbiBvbiBhbGwgdGhyZWUgbm9kZXMqKiwgY29vcmRpbmF0ZWQgdmlhIGEgc2hhcmVkIGBmbG9ja2AgKyBzaGFyZWQKICAgbG9ncm90YXRlIHN0YXRlIGZpbGUsIGJvdGggYWxzbyBvbiB0aGUgQ2VwaEZTIG1vdW50IChzZWUKICAgYHRyYWVmaWstbG9ncm90YXRlLnNoYCkuIFdoaWNoZXZlciBub2RlJ3MgY3JvbiBmaXJlcyBmaXJzdCBncmFicyB0aGUgbG9jaywKICAgcm90YXRlcyBpZiBkdWUsIGFuZCB1cGRhdGVzIHRoZSBzaGFyZWQgc3RhdGUgc28gdGhlIG90aGVyIHR3byBub2Rlcydcm9uCiAgIHJ1bnMgc2VlIGl0J3MgYWxyZWFkeSBkb25lLiBObyBzaW5nbGUgbm9kZSBpcyBhIHJvdGF0aW9uIFNQT0YuCjMuICoqU2l6ZS10cmlnZ2VyZWQqKiAoYHNpemUgMjUwTWApIHJhdGhlciB0aGFuIGNhbGVuZGFyLXRyaWdnZXJlZCAoYGRhaWx5YCkgLS0KICAgdGhpcyBmaWxlIGNhbiBncm93IGZhc3QgdW5kZXIgYnVyc3RzIChzZWUgaW5jaWRlbnQgYmFja2dyb3VuZCBhYm92ZSk7IGEKICAgcHVyZWx5IGRhaWx5IGludGVydmFsIHdvdWxkIHN0aWxsIGxldCBpdCBiYWxsb29uIGJldHdlZW4gcnVucy4gQ3JvbiBjaGVja3MKICAgZXZlcnkgMTUgbWludXRlcywgc28gaXQgY2Fubm90IGdyb3cgbXVjaCBwYXN0IHRoZSAyNTBNIHRocmVzaG9sZCBpbiBwcmFjdGljZS4KCiMjIEluc3RhbGwgKG9uZS10aW1lLCBwZXIgbm9kZSkKCk11c3QgcnVuIG9uICoqYWxsIHRocmVlKiogZG9ja2VyIExYQ3MgLS0gdGhpcyBpcyBob3N0LWxldmVsIGNyb24vbG9ncm90YXRlCmNvbmZpZywgbm90IHNvbWV0aGluZyBgc3RhY2stZGVwbG95LnNoYCBjYW4gcmVhY2ggKGl0IG9ubHkgdG91Y2hlcyBTd2FybQpzZXJ2aWNlcywgbm90IGhvc3QgY3JvbiBqb2JzKS4KCmBgYGJhc2gKIyBPbiBlYWNoIG9mIGRvY2tlci0xICgxOTIuMTY4LjQuMzEpLCBkb2NrZXItMiAoMTkyLjE2OC40LjMyKSwgZG9ja2VyLTMgKDE5Mi4xNjguNC4zMyk6CnNzaCByb290QDwxOTIuMTY4LjQuMzF8LjMyfC4zMz4KY2QgL3ZvbHVtZTEvZG9ja2VyL2NvbXBvc2UtZmlsZXMKYmFzaCBkZXBsb3kvZ2l0LWd1YXJkLnNoICAgICAgICAgIyBjb25maXJtIHN5bmMgZmlyc3QsIGFzIGFsd2F5cwpzdWRvIGJhc2ggdHJhZWZpay9sb2dyb3RhdGUvaW5zdGFsbC5zaAogYGBgCgpWZXJpZnkgb24gZWFjaCBub2RlOgpgYGBiYXNoCmNhdCAvZXRjL2Nyb24uZC90cmFlZmlrLWxvZ3JvdGF0ZQpsb2dyb3RhdGUgLWQgL2V0Yy9sb2dyb3RhdGUuZC90cmFlZmlrLWxvZ3MgICAjIGRyeS1ydW4sIGNvbmZpcm1zIHN5bnRheAogYGBgCgojIyBCb290c3RyYXBwaW5nIC0tIHNocmlua2luZyB0aGUgKmVYaXN0aW5nKiBvdmVyc2l6ZWQgbG9nIGZpbGUKCioqTm90IGRvbmUgYXV0b21hdGljYWxseSBieSBgaW5zdGFsbC5zaGAuKiogVGhlIG5ldyBzaXplLXRyaWdnZXJlZCBjb25maWcgb25seQpwcmV2ZW50cyAqZnV0dXJlKiB1bmJvdW5kZWQgZ3Jvd3RoIC0tIGl0IHdvbid0IHRvdWNoIHRoZSBjdXJyZW50IDE1LjVHQiBmaWxlCnVudGlsIHRoZSBuZXh0IHRpbWUgaXQgY3Jvc3NlcyAyNTBNIChpLmUuIG5ldmVyLCBzaW5jZSBpdCdzIGFscmVhZHkgd2VsbCBwYXN0CnRoYXQgYW5kIGxvZ3JvdGF0ZSBvbmx5IGFjdHMgb24gY3Jvc3NpbmcgdGhlIHRocmVzaG9sZCBnb2luZyBmb3J3YXJkIGZyb20gaXRzCnJlY29yZGVkIHNpemUgYXQgbGFzdCBjaGVjaykuCgpgY29weXRydW5jYXRlYCBhbHdheXMgY29waWVzIHRoZSBmdWxsIGN1cnJlbnQgZmlsZSBiZWZvcmUgdHJ1bmNhdGluZyBpdCAtLSB0aGF0J3MKaW5oZXJlbnQgdG8gaG93IGl0IHdvcmtzLCBub3QgYSBidWcuIEZvcmNpbmcgYSByb3RhdGlvbiBvZiB0aGUgY3VycmVudCAxNS41R0IKZmlsZSB3b3VsZCBtb21lbnRhcmlseSBuZWVkIHJvdWdobHkgYW5vdGhlciAxNS1HQi1zaXplZCBjaHVuayBvZiBmcmVlIHNwYWNlIC0tCmFuZCBgL3ZvbHVtZTEvZG9ja2VyLXJvb3RgIG9ubHkgaGFkICoqfjE0RyBmcmVlKiogYXQgbGFzdCBjaGVjay4gRG9pbmcgdGhpcwpibGluZGx5IGNvdWxkIHRpcCB0aGUgdm9sdW1lIHRvIDEwMCUgbWlkLW9wZXJhdGlvbi4KCioqUmVjb21tZW5kZWQgbWFudWFsIHN0ZXAgKG9wZXJhdG9yLXJ1biwgb24gYW55IG9uZSBub2RlIC0tIGl0J3MgdGhlIHNhbWUKQ2VwaEZTIGZpbGUgZnJvbSBhbGwgdGhyZWUpKio6IHRoZXNlIGFyZSBkaWFnbm9zdGljIGFjY2Vzcy9kZWJ1ZyBsb2dzLCBub3Qgc29tZXRoaW5nCndvcnRoIHByZXNlcnZpbmcgaW4gZnVsbCwgc28ganVzdCB0cnVuY2F0ZSBkaXJlY3RseSByYXRoZXIgdGhhbiBjb21wcmVzcy10aGVuLQp0cnVuY2F0ZTogCgpgYGBiYXNoCiMgT3B0aW9uYWw6IGtlZXAgYSBzbWFsbCB0YWlsIHNhbXBsZSBmb3IgcmVmZXJlbmNlIGJlZm9yZSB0cnVuY2F0aW5nCnRhaWwgLWMgNTAwMDAwMDAgL3ZvbHVtZTEvZG9ja2VyL3RyYWVmaWsvbG9ncy9hY2Nlc3MubG9nID4gXAogIC92b2x1bWUxL2RvY2tlci90cmFlZmlrLWFjY2Vzcy1sb2ctYXJjaGl2ZS0kKGRhdGUgKyVZJW0lZCkubG9nCgojIFRoZW4gdHJ1bmNhdGUgaW4gcGxhY2UgKHNhZmUgLS0gVHJhZWZpaydzIGV4aXN0aW5nIGZpbGUgaGFuZGxlcyBvbiBhbGwgMwojIG5vZGVzIHN0YXkgdmFsaWQsIHNhbWUgYXMgY29weXRydW5jYXRlJ3Mgb3duIG1lY2hhbmlzbSk6CjogPiAvdm9sdW1lMS9kb2NrZXIvdHJhZWZpay9sb2dzL2FjY2Vzcy5sb2cKOiA+IC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MvdHJhZWZpay5sb2cKCiMgQ29uZmlybToKZGYgLWggL3ZvbHVtZTEvZG9ja2VyLXJvb3QKbHMgLWxhIC92b2x1bWUxL2RvY2tlci90cmFlZmlrL2xvZ3MvCmBgYAoKQWZ0ZXIgdGhpcyBvbmUtdGltZSBib290c3RyYXAsIHRoZSBjcm9uK2xvZ3JvdGF0ZSBzZXR1cCBrZWVwcyBpdCBib3VuZGVkCihyb3RhdGVzIGF0IDI1ME0sIGtlZXBzIDQ4IGNvbXByZXNzZWQgZ2VuZXJhdGlvbnMsIHBydW5lcyBhbnl0aGluZyBvdmVyIDE0CmRheXMgb2xkKSBnb2luZyBmb3J3YXJkIHdpdGhvdXQgbmVlZGluZyBhbnkgZnVydGhlciBtYW51YWwgaW50ZXJ2ZW50aW9uLgoKIyMgUmVsYXRlZCwgTk9UIGluY2x1ZGVkIGluIHRoaXMgY2hhbmdlIChmb2xsb3ctdXAgdG8gY29uc2lkZXIgc2VwYXJhdGVseSkKCmB0cmFlZmlrLnlhbWxgIGN1cnJlbnRseSBydW5zIGAtLWxvZy5sZXZlbD1EQlVHYCAtLSB2ZXJib3NlIGRlYnVnIGxvZ2dpbmcgaW4KcHJvZHVjdGlvbiwgd2hpY2ggaXMgYSBtZWFuaW5nZnVsIGNvbnRyaWJ1dG9yIHRvIGhvdyBmYXN0IHRoZXNlIGZpbGVzIGdyb3cuCkxvd2VyaW5nIHRvIGBJTkZPYCB3b3VsZCByZWR1Y2Ugdm9sdW1lIHNpZ25pZmljYW50bHkgYnV0IHJlcXVpcmVzIGEgcmVhbAptb2RpZmljYXRpb24gK3JlZGVwbG95IG9mIHRoZSBoaWdoLWJsYXN0LXJhZGl1cyBgdHJhZWZpa2Agc3RhY2sgKGFsbCBIVFRQL0hUVFBTCApyb3V0aW5nIGRlcGVuZHMgb24gaXQpLCBzbyBpdCdzIGludGVudGlvbmFsbHkgbGVmdCBvdXQgb2YgdGhpcyBQUiBhbmQgc2hvdWxkCmJlIGl0cyBvd24gcmV2aWV3ZWQgY2hhbmdlIGlmIHdhbnRlZC4= \ No newline at end of file +# Traefik Log Rotation + +## Why this exists + +`traefik/traefik.yaml` runs Traefik with: +- `--accesslog.filePath=/traefik/logs/access.log` +- `--log.filePath=/traefik/logs/traefik.log` +- `--log.level=DEBUG` + +Neither file has any built-in rotation -- Traefik has no native rotate-on-size +nor a SIGUSR1/reopen handler. Docker's `json-file` log-driver rotation +(`max-size`/`max-file`) only applies to stdout, not to files Traefik writes +directly via `--accesslog.filePath`/`--log.filePath`. Result: `access.log` grew +to **~15.5GB unrotated** before this was caught, on a CephFS volume +(`/volume1/docker-root`) already at 82-84% used. Both the disk pressure and +the ongoing write latency of appending to a 15GB file on a network filesystem +on every request through Traefik were flagged as a real risk factor during +troubleshooting (Uptime Kuma WebSocket flapping investigation, Sep 2026). + +**Note:** during that investigation, the actual root cause of the WebSocket +flapping turned out to be Uptime Kuma monitor misconfigurations (a Postgres +monitor throwing a null-reference error, and several monitors failing TLS +validation against self-signed/internal-IP certs) -- not this log file. This +rotation fix is still worth doing as general disk/IO hygiene, just not +causally tied to that incident. + +## Architecture constraint this design accounts for + +`traefik_reverse-proxy` runs Swarm *`mode: global`* -- one instance on **each** +of docker-1, docker-2, docker-3. All three write to the **same physical file** +via the shared CephFS bind mount `/volume1/docker/traefik -> /traefik` +(identical mount, visible identically from any node). That rules out: + +- **Signal-based rotation** (classic `create` + `postrotate` sending SIGUSR1): + Traefik doesn't implement a reopen signal, and even if it did, you'd need to + signal 3 separate per-node containers in lockstep. +- **Running logrotate on just one node**: works until that node is down, then + rotation silently stops with no alert. + +So this setup uses: +1. **`copytruncate`** (see `traefik-logs.conf`) -- all three Traefik processes + keep writing to the same inode, no signaling needed. Tradeoff: a few log + lines written in the exact copy/truncate instant can be lost -- fine for + diagnostic logs. +2. **Cron on all three nodes**, coordinated via a shared `flock` + shared + logrotate state file, both also on the CephFS mount (see + `traefik-logrotate.sh`). Whichever node's cron fires first grabs the lock, + rotates if due, and updates the shared state so the other two nodes' cron + runs see it's already done. No single node is a rotation SPOF. +3. **Size-triggered** (`size 250M`) rather than calendar-triggered (`daily`) -- + this file can grow fast under bursts (see incident background above); a + purely daily interval would still let it balloon between runs. Cron checks + every 15 minutes, so it cannot grow much past the 250M threshold in practice. + +## Install (one-time, per node) + +Must run on **all three** docker LXCs -- this is host-level cron/logrotate +config, not something `stack-deploy.sh` can reach (it only touches Swarm +services, not host cron jobs). + +```bash +# On each of docker-1 (192.168.4.31), docker-2 (192.168.4.32), docker-3 (192.168.4.33): +ssh root@<192.168.4.31|.32|.33> +cd /volume1/docker/compose-files +bash deploy/git-guard.sh # confirm sync first, as always +sudo bash traefik/logrotate/install.sh +``` + +Verify on each node: +```bash +cat /etc/cron.d/traefik-logrotate +logrotate -d /etc/logrotate.d/traefik-logs # dry-run, confirms syntax +``` + +## Bootstrapping -- shrinking the *existing* oversized log file + +**Not done automatically by `install.sh`.** The new size-triggered config only +prevents *future* unbounded growth -- it won't touch the current 15.5GB file +until the next time it crosses 250M (i.e. never, since it's already well past +that and logrotate only acts on crossing the threshold going forward from its +recorded size at last check). + +`copytruncate` always copies the full current file before truncating it -- +that's inherent to how it works, not a bug. Forcing a rotation of the current +15.5GB file would momentarily need roughly another 15-GB-sized chunk of free +space -- and `/volume1/docker-root` only had **~14G free** at last check. +Doing this blindly could tip the volume to 100% mid-operation. + +**Recommended manual step (operator-run, on any one node -- it's the same +CephFS file from all three)**: these are diagnostic access/debug logs, not +something worth preserving in full, so just truncate directly rather than +compress-then-truncate: + +```bash +# Optional: keep a small tail sample for reference before truncating +tail -c 50000000 /volume1/docker/traefik/logs/access.log > \ + /volume1/docker/traefik-access-log-archive-$(date +%Y%m%d).log + +# Then truncate in place (safe -- Traefik's existing file handles on all 3 +# nodes stay valid, same as copytruncate's own mechanism): +: > /volume1/docker/traefik/logs/access.log +: > /volume1/docker/traefik/logs/traefik.log + +# Confirm: +df -h /volume1/docker-root +ls -la /volume1/docker/traefik/logs/ +``` + +After this one-time bootstrap, the cron+logrotate setup keeps it bounded +(rotates at 250M, keeps 48 compressed generations, prunes anything over 14 +days old) going forward without needing any further manual intervention. + +## Related, NOT included in this change (follow-up to consider separately) + +`traefik.yaml` currently runs `--log.level=DEBUG` -- verbose debug logging in +production, which is a meaningful contributor to how fast these files grow. +Lowering to `INFO` would reduce volume significantly but requires a real +modification + redeploy of the high-blast-radius `traefik` stack (all +HTTP/HTTPS routing depends on it), so it's intentionally left out of this PR +and should be its own reviewed change if wanted.