This causes /volume1/docker/letsencrypt/certs/{certs,private}/* to be rewritten every ~5 seconds, which is believed to have caused a transient HA startup failure (Could not read SSL certificate ... [SSL] PEM lib) by catching the cert file mid-rewrite.
Investigation done so far
Inspected acme.json directly (read-only, redacted — no key material was printed).
All 53 certificate entries in .letsencrypt.Certificates[] have well-formed, non-empty key fields.
Decoded every entry's base64 key field and confirmed each one is a valid PEM private key (-----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----), including home.bryanmail.net (entry #39).
The Account.PrivateKey field is also a well-formed non-empty string.
Conclusion: acme.json itself is healthy. No corrupt/empty key entry found anywhere in the file.
Root cause hypothesis (unconfirmed)
Version mismatch between the two services that both read acme.json:
traefik_reverse-proxy runs public.ecr.aws/docker/library/traefik:latest — an unpinned floating tag (see separate issue for pinning this).
traefik_traefik-certs-dumper's live running container was on v2.10.0, while traefik/traefik.yaml in this repo already had it pinned to v2.11.4 — meaning the live instance was running an older image than what's committed.
Traefik's ACME storage schema has changed across major versions; a mismatch between what version of Traefik wrote acme.json and what version of traefik-certs-dumper is reading it is a plausible explanation for a decode-time panic that doesn't show up as a "bad value" under manual inspection.
Next steps
Redeploy the traefik stack (now that the unrelated YAML quoting bug blocking mount-guard.py is fixed) so the live traefik-certs-dumper actually picks up the already-pinned v2.11.4 image.
Watch docker service logs traefik_traefik-certs-dumper after redeploy — confirm whether the crash loop stops.
If it persists on v2.11.4, check the actual live Traefik version (docker exec <reverse-proxy> traefik version) and compare against traefik-certs-dumper's supported schema versions; may need to pin both to a known-compatible pair.
## Summary
`traefik_traefik-certs-dumper` is crash-looping (restart policy `any` / 5s delay), panicking on every startup attempt with:
```
panic: unsupported key type: ''
.../dumper/v3.extractPEMPrivateKey(...)
.../dumper/v3.Dump(...)
```
This causes `/volume1/docker/letsencrypt/certs/{certs,private}/*` to be rewritten every ~5 seconds, which is believed to have caused a transient HA startup failure (`Could not read SSL certificate ... [SSL] PEM lib`) by catching the cert file mid-rewrite.
## Investigation done so far
- Inspected `acme.json` directly (read-only, redacted — no key material was printed).
- All 53 certificate entries in `.letsencrypt.Certificates[]` have well-formed, non-empty `key` fields.
- Decoded every entry's base64 `key` field and confirmed each one is a valid PEM private key (`-----BEGIN PRIVATE KEY-----` or `-----BEGIN RSA PRIVATE KEY-----`), including `home.bryanmail.net` (entry #39).
- The `Account.PrivateKey` field is also a well-formed non-empty string.
- **Conclusion: `acme.json` itself is healthy.** No corrupt/empty key entry found anywhere in the file.
## Root cause hypothesis (unconfirmed)
Version mismatch between the two services that both read `acme.json`:
- `traefik_reverse-proxy` runs `public.ecr.aws/docker/library/traefik:latest` — an unpinned floating tag (see separate issue for pinning this).
- `traefik_traefik-certs-dumper`'s **live running container** was on `v2.10.0`, while `traefik/traefik.yaml` in this repo already had it pinned to `v2.11.4` — meaning the live instance was running an older image than what's committed.
Traefik's ACME storage schema has changed across major versions; a mismatch between what version of Traefik wrote `acme.json` and what version of `traefik-certs-dumper` is reading it is a plausible explanation for a decode-time panic that doesn't show up as a "bad value" under manual inspection.
## Next steps
1. Redeploy the `traefik` stack (now that the unrelated YAML quoting bug blocking `mount-guard.py` is fixed) so the live `traefik-certs-dumper` actually picks up the already-pinned `v2.11.4` image.
2. Watch `docker service logs traefik_traefik-certs-dumper` after redeploy — confirm whether the crash loop stops.
3. If it persists on `v2.11.4`, check the actual live Traefik version (`docker exec <reverse-proxy> traefik version`) and compare against `traefik-certs-dumper`'s supported schema versions; may need to pin both to a known-compatible pair.
traefik_traefik-certs-dumper current task: Running for 11+ minutes, no panic: entries in that window (docker service logs ... --since 5m | grep -c "panic:" → 0).
Live image confirmed: ghcr.io/ldez/traefik-certs-dumper:v2.11.4@sha256:a34b27edf8e5e96a215035a6bfba00d769d2638f8444a5dd6dc27299d75f43c5 — matches what was already pinned in traefik/traefik.yaml, previously the live instance was still on the older v2.10.0 before this redeploy.
/volume1/docker/letsencrypt/certs/certs/home.bryanmail.net.crt mtime is now stable (single timestamp, not being rewritten every ~5s as before).
The Failed/Shutdown task entries visible in docker service ps history are from the tail end of the redeploy transition itself (timestamps ~11 min ago), not an ongoing loop — no new failures since.
Root cause hypothesis (version mismatch between reverse-proxy and traefik-certs-dumper) is consistent with this — redeploying picked up the already-pinned v2.11.4 dumper image and the crash loop stopped. Closing.
Related: #11 (reverse-proxy is still on unpinned :latest) remains open — worth keeping an eye on this issue if that tag ever drifts again in the future.
## Verified resolved after traefik redeploy
Checked live state on docker-2/docker-3:
- `traefik_traefik-certs-dumper` current task: `Running` for 11+ minutes, no `panic:` entries in that window (`docker service logs ... --since 5m | grep -c "panic:"` → `0`).
- Live image confirmed: `ghcr.io/ldez/traefik-certs-dumper:v2.11.4@sha256:a34b27edf8e5e96a215035a6bfba00d769d2638f8444a5dd6dc27299d75f43c5` — matches what was already pinned in `traefik/traefik.yaml`, previously the live instance was still on the older `v2.10.0` before this redeploy.
- `/volume1/docker/letsencrypt/certs/certs/home.bryanmail.net.crt` mtime is now stable (single timestamp, not being rewritten every ~5s as before).
- The `Failed`/`Shutdown` task entries visible in `docker service ps` history are from the tail end of the redeploy transition itself (timestamps ~11 min ago), not an ongoing loop — no new failures since.
Root cause hypothesis (version mismatch between `reverse-proxy` and `traefik-certs-dumper`) is consistent with this — redeploying picked up the already-pinned `v2.11.4` dumper image and the crash loop stopped. Closing.
Related: #11 (reverse-proxy is still on unpinned `:latest`) remains open — worth keeping an eye on this issue if that tag ever drifts again in the future.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
traefik_traefik-certs-dumperis crash-looping (restart policyany/ 5s delay), panicking on every startup attempt with:This causes
/volume1/docker/letsencrypt/certs/{certs,private}/*to be rewritten every ~5 seconds, which is believed to have caused a transient HA startup failure (Could not read SSL certificate ... [SSL] PEM lib) by catching the cert file mid-rewrite.Investigation done so far
acme.jsondirectly (read-only, redacted — no key material was printed)..letsencrypt.Certificates[]have well-formed, non-emptykeyfields.keyfield and confirmed each one is a valid PEM private key (-----BEGIN PRIVATE KEY-----or-----BEGIN RSA PRIVATE KEY-----), includinghome.bryanmail.net(entry #39).Account.PrivateKeyfield is also a well-formed non-empty string.acme.jsonitself is healthy. No corrupt/empty key entry found anywhere in the file.Root cause hypothesis (unconfirmed)
Version mismatch between the two services that both read
acme.json:traefik_reverse-proxyrunspublic.ecr.aws/docker/library/traefik:latest— an unpinned floating tag (see separate issue for pinning this).traefik_traefik-certs-dumper's live running container was onv2.10.0, whiletraefik/traefik.yamlin this repo already had it pinned tov2.11.4— meaning the live instance was running an older image than what's committed.Traefik's ACME storage schema has changed across major versions; a mismatch between what version of Traefik wrote
acme.jsonand what version oftraefik-certs-dumperis reading it is a plausible explanation for a decode-time panic that doesn't show up as a "bad value" under manual inspection.Next steps
traefikstack (now that the unrelated YAML quoting bug blockingmount-guard.pyis fixed) so the livetraefik-certs-dumperactually picks up the already-pinnedv2.11.4image.docker service logs traefik_traefik-certs-dumperafter redeploy — confirm whether the crash loop stops.v2.11.4, check the actual live Traefik version (docker exec <reverse-proxy> traefik version) and compare againsttraefik-certs-dumper's supported schema versions; may need to pin both to a known-compatible pair.Verified resolved after traefik redeploy
Checked live state on docker-2/docker-3:
traefik_traefik-certs-dumpercurrent task:Runningfor 11+ minutes, nopanic:entries in that window (docker service logs ... --since 5m | grep -c "panic:"→0).ghcr.io/ldez/traefik-certs-dumper:v2.11.4@sha256:a34b27edf8e5e96a215035a6bfba00d769d2638f8444a5dd6dc27299d75f43c5— matches what was already pinned intraefik/traefik.yaml, previously the live instance was still on the olderv2.10.0before this redeploy./volume1/docker/letsencrypt/certs/certs/home.bryanmail.net.crtmtime is now stable (single timestamp, not being rewritten every ~5s as before).Failed/Shutdowntask entries visible indocker service pshistory are from the tail end of the redeploy transition itself (timestamps ~11 min ago), not an ongoing loop — no new failures since.Root cause hypothesis (version mismatch between
reverse-proxyandtraefik-certs-dumper) is consistent with this — redeploying picked up the already-pinnedv2.11.4dumper image and the crash loop stopped. Closing.Related: #11 (reverse-proxy is still on unpinned
:latest) remains open — worth keeping an eye on this issue if that tag ever drifts again in the future.