traefik/traefik.env was committed to Gitea (since c2069e3, "migrate to folder-based stack layout") with a literal placeholder as KEEPALIVED_PASSWORD:
KEEPALIVED_PASSWORD=***REDACTED***
Because the file was tracked, every git-guard resync / checkout / manual delete-and-restore brought this exact broken value back onto disk. keepalived-master's running container env showed the broken value; keepalived-backup still had a stale-but-correct value from before the bad commit landed — so the two replicas disagreed on VRRP auth, producing a continuous (VI_1) received an invalid passwd! log flood and real VIP (192.168.4.30) instability (this is what made git.bryanmail.net intermittently unreachable during this incident, since it resolves through the VIP).
Confirmed via byte-level check (length=14, base64, sha256) that the file literally contains the ASCII string ***REDACTED***, not a masked display artifact.
What this PR does
Removes traefik/traefik.env from git entirely. It should never have been tracked (.gitignore already has a blanket *.env rule, but that doesn't retroactively untrack a file committed before the rule existed).
Adds traefik/traefik.env.template — the new authoritative source, following the exact pattern already proven for the ai stack. Non-secret config (KEEPALIVED_VIRTUAL_IPS) is a literal value; KEEPALIVED_PASSWORD is a ${TRAEFIK_KEEPALIVED_PASSWORD} placeholder resolved by provision-stack.py.
Registers traefik in secrets/secrets-map.yaml (env_template / env_dest, no docker_secrets needed since this stack has exactly one secret consumed via rendered env file, not a Docker secret).
Updates .woodpecker/deploy.yml:
Adds TRAEFIK_KEEPALIVED_PASSWORD: from_secret: traefik_keepalived_password to the provision-secrets environment block.
Replaces the traefik legacy case (previously grouped under maintenance|media|unifi|guacamole|security|auth|traefik|meshcentral|ddm → "no Docker secrets, host .env only") with its own one-line case calling python3 deploy/provision-stack.py traefik, matching the ai) case exactly.
Updates secrets/traefik.secrets.example and secrets/README.md to reflect the completed migration and document the incident as a cautionary note for future stacks.
Diffed the $${ occurrence count in deploy.yml against the previous version before committing, per this file's own 2026-08-26 incident note about full-file rewrites silently dropping $ characters. No other case entries were touched.
⚠️ Action required before/after merge
Add the Woodpecker secrettraefik_keepalived_password (repo Settings → Secrets) with a new value — do not reuse ***REDACTED***. Recommend generating a fresh value ≤ 8 characters, since classic VRRP simple-auth silently truncates to 8 chars regardless of the value's real length (the previous "working" value on -backup was 30 chars, so only the first 8 were ever actually significant).
After merge + secret creation, first push will render traefik/traefik.env fresh and require a manual stack-deploy.sh traefik (traefik is bootstrap-tier — deploy is never automatic).
Verify post-deploy: docker service logs traefik_keepalived-master --tail 10 and -backup should show no "invalid passwd" errors, and both services' env should match via docker service inspect ... --format '{{range .Spec.TaskTemplate.ContainerSpec.Env}}{{println .}}{{end}}' | grep -i pass.
Blast radius
High — this is the traefik stack (entry point for all services) and touches the shared .woodpecker/deploy.yml. No live service is affected by merging this PR alone (rendering/provisioning only happens on the next push that changes traefik/, and deploy for this stack has always been manual). The actual keepalived fix only takes effect after the new secret is created andstack-deploy.sh traefik is run manually.
## Root cause
`traefik/traefik.env` was committed to Gitea (since `c2069e3`, "migrate to folder-based stack layout") with a literal placeholder as `KEEPALIVED_PASSWORD`:
```
KEEPALIVED_PASSWORD=***REDACTED***
```
Because the file was tracked, every git-guard resync / checkout / manual delete-and-restore brought this exact broken value back onto disk. `keepalived-master`'s running container env showed the broken value; `keepalived-backup` still had a stale-but-*correct* value from before the bad commit landed — so the two replicas disagreed on VRRP auth, producing a continuous `(VI_1) received an invalid passwd!` log flood and real VIP (192.168.4.30) instability (this is what made `git.bryanmail.net` intermittently unreachable during this incident, since it resolves through the VIP).
Confirmed via byte-level check (length=14, base64, sha256) that the file literally contains the ASCII string `***REDACTED***`, not a masked display artifact.
## What this PR does
1. **Removes `traefik/traefik.env` from git entirely.** It should never have been tracked (`.gitignore` already has a blanket `*.env` rule, but that doesn't retroactively untrack a file committed before the rule existed).
2. **Adds `traefik/traefik.env.template`** — the new authoritative source, following the exact pattern already proven for the `ai` stack. Non-secret config (`KEEPALIVED_VIRTUAL_IPS`) is a literal value; `KEEPALIVED_PASSWORD` is a `${TRAEFIK_KEEPALIVED_PASSWORD}` placeholder resolved by `provision-stack.py`.
3. **Registers `traefik` in `secrets/secrets-map.yaml`** (`env_template` / `env_dest`, no `docker_secrets` needed since this stack has exactly one secret consumed via rendered env file, not a Docker secret).
4. **Updates `.woodpecker/deploy.yml`:**
- Adds `TRAEFIK_KEEPALIVED_PASSWORD: from_secret: traefik_keepalived_password` to the `provision-secrets` environment block.
- Replaces the `traefik` legacy case (previously grouped under `maintenance|media|unifi|guacamole|security|auth|traefik|meshcentral|ddm` → "no Docker secrets, host .env only") with its own one-line case calling `python3 deploy/provision-stack.py traefik`, matching the `ai)` case exactly.
5. **Updates `secrets/traefik.secrets.example` and `secrets/README.md`** to reflect the completed migration and document the incident as a cautionary note for future stacks.
Diffed the `$${` occurrence count in `deploy.yml` against the previous version before committing, per this file's own 2026-08-26 incident note about full-file rewrites silently dropping `$` characters. No other case entries were touched.
## ⚠️ Action required before/after merge
- **Add the Woodpecker secret** `traefik_keepalived_password` (repo Settings → Secrets) with a **new** value — do not reuse `***REDACTED***`. Recommend generating a fresh value ≤ 8 characters, since classic VRRP simple-auth silently truncates to 8 chars regardless of the value's real length (the previous "working" value on `-backup` was 30 chars, so only the first 8 were ever actually significant).
- After merge + secret creation, first push will render `traefik/traefik.env` fresh and require a manual `stack-deploy.sh traefik` (traefik is bootstrap-tier — deploy is never automatic).
- Verify post-deploy: `docker service logs traefik_keepalived-master --tail 10` and `-backup` should show no "invalid passwd" errors, and both services' env should match via `docker service inspect ... --format '{{range .Spec.TaskTemplate.ContainerSpec.Env}}{{println .}}{{end}}' | grep -i pass`.
## Blast radius
High — this is the `traefik` stack (entry point for all services) and touches the shared `.woodpecker/deploy.yml`. No live service is affected by merging this PR alone (rendering/provisioning only happens on the next push that changes `traefik/`, and deploy for this stack has always been manual). The actual keepalived fix only takes effect after the new secret is created **and** `stack-deploy.sh traefik` is run manually.
The committed version contained a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD, which was restored onto disk every time the real file
was deleted or the local checkout re-synced from git. This caused VRRP auth
to mismatch between keepalived-master and keepalived-backup (master picked
up the broken committed value, backup retained a stale-but-correct value),
producing a continuous "(VI_1) received an invalid passwd!" flood and VIP
instability. traefik.env is now rendered fresh from traefik.env.template by
deploy/provision-stack.py on every provisioning run and never committed.
Root cause of the 2026-09-12 VIP instability incident: traefik/traefik.env
was committed to git with a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD. Every git-guard resync/checkout restored that broken
value onto disk, diverging from keepalived-backup's stale-but-correct
in-memory value and causing a continuous VRRP auth failure + VIP flapping.
- Adds TRAEFIK_KEEPALIVED_PASSWORD (from_secret: traefik_keepalived_password)
to provision-secrets environment block
- Replaces the traefik legacy case-entry (previously grouped with
maintenance|media|unifi|... as "no Docker secrets, host .env only") with
a call to deploy/provision-stack.py traefik, matching the ai) stack's
manifest-driven pattern
- traefik/traefik.env is now rendered fresh from traefik/traefik.env.template
on every provisioning run and never committed to git again
Diffed $${ occurrence count against previous version before committing,
per this file's own 2026-08-26 incident note.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Root cause
traefik/traefik.envwas committed to Gitea (sincec2069e3, "migrate to folder-based stack layout") with a literal placeholder asKEEPALIVED_PASSWORD:Because the file was tracked, every git-guard resync / checkout / manual delete-and-restore brought this exact broken value back onto disk.
keepalived-master's running container env showed the broken value;keepalived-backupstill had a stale-but-correct value from before the bad commit landed — so the two replicas disagreed on VRRP auth, producing a continuous(VI_1) received an invalid passwd!log flood and real VIP (192.168.4.30) instability (this is what madegit.bryanmail.netintermittently unreachable during this incident, since it resolves through the VIP).Confirmed via byte-level check (length=14, base64, sha256) that the file literally contains the ASCII string
***REDACTED***, not a masked display artifact.What this PR does
traefik/traefik.envfrom git entirely. It should never have been tracked (.gitignorealready has a blanket*.envrule, but that doesn't retroactively untrack a file committed before the rule existed).traefik/traefik.env.template— the new authoritative source, following the exact pattern already proven for theaistack. Non-secret config (KEEPALIVED_VIRTUAL_IPS) is a literal value;KEEPALIVED_PASSWORDis a${TRAEFIK_KEEPALIVED_PASSWORD}placeholder resolved byprovision-stack.py.traefikinsecrets/secrets-map.yaml(env_template/env_dest, nodocker_secretsneeded since this stack has exactly one secret consumed via rendered env file, not a Docker secret)..woodpecker/deploy.yml:TRAEFIK_KEEPALIVED_PASSWORD: from_secret: traefik_keepalived_passwordto theprovision-secretsenvironment block.traefiklegacy case (previously grouped undermaintenance|media|unifi|guacamole|security|auth|traefik|meshcentral|ddm→ "no Docker secrets, host .env only") with its own one-line case callingpython3 deploy/provision-stack.py traefik, matching theai)case exactly.secrets/traefik.secrets.exampleandsecrets/README.mdto reflect the completed migration and document the incident as a cautionary note for future stacks.Diffed the
$${occurrence count indeploy.ymlagainst the previous version before committing, per this file's own 2026-08-26 incident note about full-file rewrites silently dropping$characters. No other case entries were touched.⚠️ Action required before/after merge
traefik_keepalived_password(repo Settings → Secrets) with a new value — do not reuse***REDACTED***. Recommend generating a fresh value ≤ 8 characters, since classic VRRP simple-auth silently truncates to 8 chars regardless of the value's real length (the previous "working" value on-backupwas 30 chars, so only the first 8 were ever actually significant).traefik/traefik.envfresh and require a manualstack-deploy.sh traefik(traefik is bootstrap-tier — deploy is never automatic).docker service logs traefik_keepalived-master --tail 10and-backupshould show no "invalid passwd" errors, and both services' env should match viadocker service inspect ... --format '{{range .Spec.TaskTemplate.ContainerSpec.Env}}{{println .}}{{end}}' | grep -i pass.Blast radius
High — this is the
traefikstack (entry point for all services) and touches the shared.woodpecker/deploy.yml. No live service is affected by merging this PR alone (rendering/provisioning only happens on the next push that changestraefik/, and deploy for this stack has always been manual). The actual keepalived fix only takes effect after the new secret is created andstack-deploy.sh traefikis run manually.Root cause of the 2026-09-12 VIP instability incident: traefik/traefik.env was committed to git with a literal "***REDACTED***" placeholder as KEEPALIVED_PASSWORD. Every git-guard resync/checkout restored that broken value onto disk, diverging from keepalived-backup's stale-but-correct in-memory value and causing a continuous VRRP auth failure + VIP flapping. - Adds TRAEFIK_KEEPALIVED_PASSWORD (from_secret: traefik_keepalived_password) to provision-secrets environment block - Replaces the traefik legacy case-entry (previously grouped with maintenance|media|unifi|... as "no Docker secrets, host .env only") with a call to deploy/provision-stack.py traefik, matching the ai) stack's manifest-driven pattern - traefik/traefik.env is now rendered fresh from traefik/traefik.env.template on every provisioning run and never committed to git again Diffed $${ occurrence count against previous version before committing, per this file's own 2026-08-26 incident note.