Author SHA1 Message Date
Bot 9ec1b1811b feat(stack-deploy): accept -e/--emergency and -f/--force flags, pass through to git-guard.sh
Flags may appear before or after the stack name (stack-deploy.sh traefik -e
and stack-deploy.sh -e traefik both work) — the stack name is whatever
argument isn't a recognized flag. No change to the render/mount-guard/deploy
logic; this only changes how the git-guard.sh pre-flight is invoked. See
deploy/git-guard.sh header for what each flag actually does.
2026-09-12 00:04:42 -07:00
Bot 8fa085ebde feat(git-guard): add -e/--emergency (try alternate Gitea hosts) and -f/--force (skip sync check)
Adds two opt-in flags to break the circular dependency discovered during
the 2026-09-12 traefik/keepalived incident: Traefik down -> VIP/hostname
unreachable -> git-guard can't fetch -> stack-deploy.sh traefik blocked ->
can't redeploy traefik to fix Traefik.

-e/--emergency: probes git.bryanmail.net, then the VIP (192.168.4.30), then
each node's direct IP (.31/.32/.33) on Gitea's direct ingress port 3000
(bypasses Traefik/VIP entirely), switches origin to the first reachable one,
then runs the NORMAL sync logic against it (behind/ahead/diverged handling
unchanged — this only changes which host is used, never skips the safety
checks). Prints a reminder to restore the real origin URL afterward; never
persists the swap anywhere.

-f/--force: skips the sync check entirely, for the genuine last-resort case
where Gitea itself (not just routing) is unreachable. Loud warning banner.

Verified git ls-remote succeeds unauthenticated over plain http against
each candidate before writing this, so no credential-smuggling concern.
2026-09-12 00:03:59 -07:00
Bot 5ab1ae1e8b chore(traefik): re-trigger provision-secrets — traefik.env was found missing on disk post-migration
ci/woodpecker/push/deploy Pipeline was successful
traefik/traefik.env did not exist on the host after PR #21 merged. Root
cause: provisioning only renders on a push touching traefik/, and traefik
is bootstrap-tier (stack-deploy.sh runs manually, never via the pipeline).
A manual stack-deploy.sh traefik run apparently happened while the env file
was absent, causing Swarm's own interpolation to render
KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS as empty strings on
keepalived-master. This comment-only change re-triggers provision-secrets;
a manual stack-deploy.sh traefik run is still required afterward to apply
the freshly-rendered env to the running services.
2026-09-11 23:17:13 -07:00
AVB afc96a6d37 Merge pull request 'fix(traefik): migrate KEEPALIVED_PASSWORD to Pattern C (manifest-driven), stop committing traefik.env' (#21) from migrate-traefik-keepalived-secret into main
ci/woodpecker/push/deploy Pipeline was successful
Reviewed-on: #21
https://ai.bryanmail.net/c/7440e9d5-7e39-40f9-a35a-33016b6ae8d1
2026-09-11 22:52:35 -07:00
Bot cd42e35265 docs(secrets): mark traefik migration complete, add committed-env-file warning 2026-09-11 22:42:06 -07:00
Bot 2a3f2bdd1e docs(traefik): update secrets reference for manifest-driven Pattern C migration 2026-09-11 22:42:04 -07:00
Bot d9f151a1c3 feat(traefik): migrate keepalived password to manifest-driven Pattern C provisioning
Root cause of the 2026-09-12 VIP instability incident: traefik/traefik.env
was committed to git with a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD. Every git-guard resync/checkout restored that broken
value onto disk, diverging from keepalived-backup's stale-but-correct
in-memory value and causing a continuous VRRP auth failure + VIP flapping.

- Adds TRAEFIK_KEEPALIVED_PASSWORD (from_secret: traefik_keepalived_password)
  to provision-secrets environment block
- Replaces the traefik legacy case-entry (previously grouped with
  maintenance|media|unifi|... as "no Docker secrets, host .env only") with
  a call to deploy/provision-stack.py traefik, matching the ai) stack's
  manifest-driven pattern
- traefik/traefik.env is now rendered fresh from traefik/traefik.env.template
  on every provisioning run and never committed to git again

Diffed $${ occurrence count against previous version before committing,
per this file's own 2026-08-26 incident note.
2026-09-11 22:40:36 -07:00
Bot 201d8418ee feat(traefik): register traefik in manifest-driven secrets provisioning 2026-09-11 22:38:39 -07:00
Bot 5edebd977d fix(traefik): remove tracked traefik.env — never commit secret-bearing env files
The committed version contained a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD, which was restored onto disk every time the real file
was deleted or the local checkout re-synced from git. This caused VRRP auth
to mismatch between keepalived-master and keepalived-backup (master picked
up the broken committed value, backup retained a stale-but-correct value),
producing a continuous "(VI_1) received an invalid passwd!" flood and VIP
instability. traefik.env is now rendered fresh from traefik.env.template by
deploy/provision-stack.py on every provisioning run and never committed.
2026-09-11 22:38:39 -07:00
Bot ffec3ea5b7 feat(traefik): add authoritative env template for manifest-driven provisioning 2026-09-11 22:38:38 -07:00
8 changed files with 286 additions and 35 deletions
+27 -1
View File
@@ -70,6 +70,16 @@ when:
# tolerating a failing stack-ps in verify (|| true) so a genuinely missing # tolerating a failing stack-ps in verify (|| true) so a genuinely missing
# stack produces the designed WARNING instead of killing the step. This # stack produces the designed WARNING instead of killing the step. This
# hazard was first flagged in July (PR #3, closed unmerged). # hazard was first flagged in July (PR #3, closed unmerged).
#
# 2026-09-12 MIGRATION: traefik moved from Pattern B (secrets hand-typed
# into the host-only traefik.env, no CI involvement at all) to the same
# manifest-driven provisioning as ai) below. Root cause of the migration:
# traefik.env had been accidentally committed to git with a literal
# "***REDACTED***" placeholder as KEEPALIVED_PASSWORD; every git-guard
# resync/checkout silently restored that broken value, which diverged from
# keepalived-backup's stale-but-correct in-memory value and produced a
# continuous VRRP auth failure ("received an invalid passwd!") and VIP
# instability. The manifest-driven path never commits the rendered file.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
steps: steps:
@@ -216,6 +226,11 @@ steps:
from_secret: flowagent_azure_tenant_id from_secret: flowagent_azure_tenant_id
FLOWAGENT_AZURE_CLIENT_SECRET: FLOWAGENT_AZURE_CLIENT_SECRET:
from_secret: flowagent_azure_client_secret from_secret: flowagent_azure_client_secret
# ── traefik stack (manifest-driven — secrets/secrets-map.yaml +
# traefik/traefik.env.template + deploy/provision-stack.py).
# Migrated 2026-09-12; see header note above for root cause. ──
TRAEFIK_KEEPALIVED_PASSWORD:
from_secret: traefik_keepalived_password
commands: commands:
- apk add --no-cache openssh-client python3 py3-yaml - apk add --no-cache openssh-client python3 py3-yaml
- mkdir -p ~/.ssh - mkdir -p ~/.ssh
@@ -243,8 +258,19 @@ steps:
for STACK in $ALL_STACKS; do for STACK in $ALL_STACKS; do
echo "Provisioning: $STACK" echo "Provisioning: $STACK"
case "$STACK" in case "$STACK" in
maintenance|media|unifi|guacamole|security|auth|traefik|meshcentral|ddm) maintenance|media|unifi|guacamole|security|auth|meshcentral|ddm)
echo " No Docker secrets for $STACK — secrets in host .env";; echo " No Docker secrets for $STACK — secrets in host .env";;
traefik)
# MIGRATED (2026-09-12) to manifest-driven provisioning after
# discovering traefik.env was committed to git with a literal
# "***REDACTED***" placeholder as KEEPALIVED_PASSWORD — every
# git-guard resync/checkout restored the broken value, causing
# VRRP auth mismatch between keepalived-master/-backup and VIP
# instability. All logic lives in deploy/provision-stack.py;
# the authoritative key list lives in
# traefik/traefik.env.template; the mapping lives in
# secrets/secrets-map.yaml. This case is intentionally one line.
python3 deploy/provision-stack.py traefik;;
immich) immich)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'immich_db_password' '$${IMMICH_DB_PASSWORD}' create_or_update_secret 'immich_db_password' '$${IMMICH_DB_PASSWORD}'
+113
View File
@@ -23,18 +23,131 @@
# remote both moved) backup/stash/reset recovery steps and exits. # remote both moved) backup/stash/reset recovery steps and exits.
# #
# Exit codes: 0 = safe to deploy, 1 = blocked, needs human intervention # Exit codes: 0 = safe to deploy, 1 = blocked, needs human intervention
#
# ── Flags (added 2026-09-12, circular-dependency bootstrap incident) ────────
#
# -e, --emergency
# Before the normal fetch, tries each candidate Gitea endpoint in order
# (git.bryanmail.net -> 192.168.4.30 VIP -> .31 -> .32 -> .33 node IPs,
# each over plain http on port 3000, which is Gitea's direct ingress
# port — bypasses Traefik/VIP entirely) and switches `origin` to the
# first one that responds to `git ls-remote` within a short timeout.
# This solves the bootstrap circular dependency where Traefik is down,
# so HTTPS access to git.bryanmail.net is unreachable, so git-guard
# can't fetch, so stack-deploy.sh can't redeploy traefik to fix itself.
# Once a reachable endpoint is found, NORMAL sync logic still runs
# (behind/ahead/diverged handling is unchanged) — this flag only changes
# *which host* is used, never skips the safety checks themselves.
# Prints a loud reminder to restore the real origin URL afterward; never
# commits the swapped URL anywhere.
#
# -f, --force
# Skips the sync check ENTIRELY — no fetch, no comparison, no commit/push
# offer. Deploys whatever is on disk right now, as-is. This is the last
# resort for a genuine emergency where NONE of the candidate hosts in
# --emergency are reachable (e.g. Gitea itself is down, not just
# routing). Prints a loud warning banner. Never use this for routine
# work — it defeats the entire purpose of this script and is the exact
# failure mode (deploying a stale/unreviewed tree) git-guard exists to
# prevent.
#
# Both flags are passed through from stack-deploy.sh's own -e/-f flags;
# see that script's header for the calling convention.
# ─────────────────────────────────────────────────────────────────────────
set -euo pipefail set -euo pipefail
DIR="/volume1/docker/compose-files" DIR="/volume1/docker/compose-files"
cd "$DIR" cd "$DIR"
# ---- Flag parsing ----
EMERGENCY=0
FORCE=0
for arg in "$@"; do
case "$arg" in
-e|--emergency) EMERGENCY=1 ;;
-f|--force) FORCE=1 ;;
esac
done
# ---- Force mode: skip everything ----
if [ "$FORCE" -eq 1 ]; then
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!! FORCE MODE (-f/--force): git-guard sync check SKIPPED ENTIRELY."
echo "!! Deploying whatever is on disk right now, as-is. No fetch, no"
echo "!! comparison with origin/main was performed. This is a LAST RESORT"
echo "!! for emergencies where origin is completely unreachable — verify"
echo "!! independently that the local tree is what you intend to deploy."
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
exit 0
fi
# Non-interactive detection (Woodpecker/cron have no TTY on stdin) # Non-interactive detection (Woodpecker/cron have no TTY on stdin)
INTERACTIVE=0 INTERACTIVE=0
[ -t 0 ] && INTERACTIVE=1 [ -t 0 ] && INTERACTIVE=1
echo "==> git-guard: checking repo sync state" echo "==> git-guard: checking repo sync state"
# ---- Emergency mode: find a reachable Gitea endpoint before fetching ----
# Candidate order: public hostname (normal path) -> VIP -> each node's direct
# IP. Each is tried over plain http on port 3000 (Gitea's direct ingress
# port, published outside Traefik — see traefik/traefik.yaml git service
# port mapping), since the whole point is to bypass Traefik/VIP when THOSE
# are what's broken. A short `git ls-remote` timeout keeps an unreachable
# candidate from stalling the whole check for long.
if [ "$EMERGENCY" -eq 1 ]; then
echo "!! EMERGENCY MODE (-e/--emergency): probing candidate Gitea endpoints"
echo "!! (bypassing the normal https://git.bryanmail.net path if needed)..."
ORIGINAL_URL="$(git remote get-url origin)"
CANDIDATES=(
"https://git.bryanmail.net/homelab/compose-files.git"
"http://192.168.4.30:3000/homelab/compose-files.git"
"http://192.168.4.31:3000/homelab/compose-files.git"
"http://192.168.4.32:3000/homelab/compose-files.git"
"http://192.168.4.33:3000/homelab/compose-files.git"
)
FOUND=""
for candidate in "${CANDIDATES[@]}"; do
echo -n " trying $candidate ... "
if timeout 5 git ls-remote "$candidate" HEAD >/dev/null 2>&1; then
echo "OK"
FOUND="$candidate"
break
else
echo "unreachable"
fi
done
if [ -z "$FOUND" ]; then
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!! EMERGENCY MODE: none of the candidate endpoints responded."
echo "!! Gitea itself may be down (not just routing) — this is beyond what"
echo "!! an alternate host path can fix. Options:"
echo "!! A) Diagnose Gitea directly: check the git_gitea-server service"
echo "!! and container on docker-1."
echo "!! B) If you are certain the on-disk tree is correct and Gitea is"
echo "!! genuinely unreachable, re-run with -f/--force instead — but"
echo "!! read that flag's warning carefully first."
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
exit 1
fi
if [ "$FOUND" != "$ORIGINAL_URL" ]; then
git remote set-url origin "$FOUND"
echo "==> origin temporarily switched to: $FOUND"
echo "!! REMINDER: once the normal path (git.bryanmail.net / Traefik) is"
echo "!! confirmed healthy again, restore the real origin URL:"
echo "!! cd $DIR && git remote set-url origin \"$ORIGINAL_URL\""
echo "!! This swap is never committed anywhere and only affects this"
echo "!! local checkout's git config."
else
echo "==> Normal origin URL ($FOUND) is reachable — no swap needed."
fi
echo
fi
git fetch origin --quiet git fetch origin --quiet
LOCAL="$(git rev-parse main)" LOCAL="$(git rev-parse main)"
+28 -2
View File
@@ -38,10 +38,36 @@
# pre-escaping) or values get doubled twice. Getting this wrong # pre-escaping) or values get doubled twice. Getting this wrong
# silently corrupts any secret/hash containing '$' (confirmed impact: # silently corrupts any secret/hash containing '$' (confirmed impact:
# LITELLM keys truncated, IMMICH_KIOSK_BASICAUTH bcrypt hash mismatched). # LITELLM keys truncated, IMMICH_KIOSK_BASICAUTH bcrypt hash mismatched).
#
# Usage: stack-deploy.sh [-e|--emergency] [-f|--force] <stack-name>
# Flags may appear before or after the stack name, e.g. both
# `stack-deploy.sh traefik -e` and `stack-deploy.sh -e traefik` work.
#
# -e/--emergency and -f/--force (added 2026-09-12, circular-dependency
# bootstrap incident — see deploy/git-guard.sh header for full detail):
# Both are passed straight through to git-guard.sh unchanged; this script
# does not interpret them itself beyond stripping them from the stack-name
# argument list. -e tries alternate Gitea endpoints (VIP, then each node's
# direct IP) before falling back to normal sync logic against whichever
# one responds. -f skips the sync check entirely — last resort only, read
# the warning banner it prints. Neither flag changes anything about the
# render/mount-guard/deploy steps below; they only affect whether and how
# git-guard.sh's pre-flight check runs.
set -euo pipefail set -euo pipefail
STACK="${1:?Usage: stack-deploy.sh <stack-name>}" # ── Flag parsing (stack name is whatever's left after flags are stripped) ──
GUARD_FLAGS=()
STACK=""
for arg in "$@"; do
case "$arg" in
-e|--emergency) GUARD_FLAGS+=(-e) ;;
-f|--force) GUARD_FLAGS+=(-f) ;;
*) STACK="$arg" ;;
esac
done
: "${STACK:?Usage: stack-deploy.sh [-e|--emergency] [-f|--force] <stack-name>}"
DIR="/volume1/docker/compose-files" DIR="/volume1/docker/compose-files"
PY="$DIR/deploy/envparse.py" PY="$DIR/deploy/envparse.py"
MOUNT_GUARD="$DIR/deploy/mount-guard.py" MOUNT_GUARD="$DIR/deploy/mount-guard.py"
@@ -50,7 +76,7 @@ GLOBAL_ENV="$DIR/deploy/global.env"
# ── Pre-flight: ensure local checkout is in sync with Gitea ───────────────── # ── Pre-flight: ensure local checkout is in sync with Gitea ─────────────────
# Prevents deploying from a stale/diverged local tree (see incident 2026-08-26). # Prevents deploying from a stale/diverged local tree (see incident 2026-08-26).
# Invoked via `bash` explicitly so the tracked file's exec bit doesn't matter. # Invoked via `bash` explicitly so the tracked file's exec bit doesn't matter.
bash "$DIR/deploy/git-guard.sh" || { echo "ERROR: git-guard check failed. Deploy aborted."; exit 1; } bash "$DIR/deploy/git-guard.sh" "${GUARD_FLAGS[@]:-}" || { echo "ERROR: git-guard check failed. Deploy aborted."; exit 1; }
# ── Locate compose file(s) ────────────────────────────────────────────────── # ── Locate compose file(s) ──────────────────────────────────────────────────
+18 -5
View File
@@ -6,9 +6,9 @@ This directory documents the secrets required for each Docker Swarm stack.
``` ```
Secret values live in Woodpecker (encrypted) Secret values live in Woodpecker (encrypted)
pipeline reads them at deploy time pipeline reads them at deploy time
Docker Swarm secret store (encrypted Raft DB, replicated across all nodes) Docker Swarm secret store (encrypted Raft DB, replicated across all nodes)
mounted into containers at runtime mounted into containers at runtime
/run/secrets/<secret_name> /run/secrets/<secret_name>
``` ```
@@ -38,7 +38,10 @@ Use the stack's `.secrets.example` file as your checklist.
### Step 2 — Add the stack's case to `.woodpecker.yml` ### Step 2 — Add the stack's case to `.woodpecker.yml`
In the `provision-secrets` step, add a case for the stack that calls In the `provision-secrets` step, add a case for the stack that calls
`create_or_update_secret` for each secret. `create_or_update_secret` for each secret. (Or, preferred for new
migrations: add an entry to `secrets/secrets-map.yaml` + a
`<stack>.env.template` and call `deploy/provision-stack.py <stack>`
instead — see the `ai` and `traefik` entries for the current pattern.)
### Step 3 — Test by pushing a trivial change to the stack's yaml file ### Step 3 — Test by pushing a trivial change to the stack's yaml file
Watch the pipeline run: provision-secrets → validate → deploy → verify → notify. Watch the pipeline run: provision-secrets → validate → deploy → verify → notify.
@@ -71,10 +74,20 @@ Examples:
- **PostgreSQL is highest risk.** Its master password is used by nearly every - **PostgreSQL is highest risk.** Its master password is used by nearly every
other stack. Migrate it last. other stack. Migrate it last.
- **Never commit a rendered env file, even by accident.** `traefik/traefik.env`
was committed to git for a period (discovered/fixed 2026-09-12) with a
literal "***REDACTED***" placeholder as KEEPALIVED_PASSWORD, which was
silently restored every time the file was deleted or the checkout resynced
from git — causing a real VRRP auth outage. `.gitignore` blanket-excludes
`*.env`, but that rule does NOT retroactively untrack a file already
committed before the rule existed. If you ever see a stack's `.env` file
show up in `git status` as tracked, stop and untrack it (`git rm --cached`)
before doing anything else.
## Migration Status ## Migration Status
| Stack | Secrets in Woodpecker | Pipeline Step Added | .env Removed | | Stack | Secrets in Woodpecker | Pipeline Step Added | .env Removed |
|-------|----------------------|---------------------|--------------| |-------|------------------------|----------------------|---------------|
| 3dprint | ⏳ | ⏳ | ⏳ | | 3dprint | ⏳ | ⏳ | ⏳ |
| ai | ⏳ | ⏳ | ⏳ | | ai | ⏳ | ⏳ | ⏳ |
| auth | ⏳ | ⏳ | ⏳ | | auth | ⏳ | ⏳ | ⏳ |
@@ -91,7 +104,7 @@ Examples:
| postgresql | ⏳ | ⏳ | ⏳ | | postgresql | ⏳ | ⏳ | ⏳ |
| productivity | ⏳ | ⏳ | ⏳ | | productivity | ⏳ | ⏳ | ⏳ |
| security | ⏳ | ⏳ | ⏳ | | security | ⏳ | ⏳ | ⏳ |
| traefik | ⏳ | ⏳ | ⏳ | | traefik | ✅ (manifest-driven, 2026-09-12) | ✅ (manifest-driven, 2026-09-12) | ⏳ |
| unifi | ✅ N/A (no secrets) | ✅ N/A | ⏳ | | unifi | ✅ N/A (no secrets) | ✅ N/A | ⏳ |
| vaultwarden | ⏳ | ⏳ | ⏳ | | vaultwarden | ⏳ | ⏳ | ⏳ |
| woodpecker | ⏳ Manual only | ⏳ N/A | ⏳ | | woodpecker | ⏳ Manual only | ⏳ N/A | ⏳ |
+15 -12
View File
@@ -1,22 +1,22 @@
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────
# secrets-map.yaml — DATA-ONLY manifest for deploy/provision-stack.py # secrets-map.yaml — DATA-ONLY manifest for deploy/provision-stack.py
# #
# RULES: # RULES:
# - This file contains NO code, NO shell, NO secret values — only names. # - This file contains NO code, NO shell, NO secret values — only names.
# - Each stack entry declares: # - Each stack entry declares:
# env_template: repo path of the FULL env-file template (tracked). # env_template: repo path of the FULL env-file template (tracked).
# The template is authoritative: the COMPLETE env file is # The template is authoritative: the COMPLETE env file is
# rendered from it on every provisioning run. Nothing is # rendered from it on every provisioning run. Nothing is
# line-edited in place, so keys can never silently go # line-edited in place, so keys can never silently go
# missing. # missing.
# env_dest: host path (relative to /volume1/docker/compose-files/) # env_dest: host path (relative to /volume1/docker/compose-files/)
# the rendered env file is shipped to. Rendered file # the rendered env file is shipped to. Rendered file
# exists ONLY on the host — never committed to git. # exists ONLY on the host — never committed to git.
# docker_secrets: map of docker-swarm-secret-name -> CI ENV VAR NAME # docker_secrets: map of docker-swarm-secret-name -> CI ENV VAR NAME
# (Pattern C). The env var must be declared via # (Pattern C). The env var must be declared via
# from_secret: in .woodpecker/deploy.yml's # from_secret: in .woodpecker/deploy.yml's
# provision-secrets step (Woodpecker v3 requires explicit # provision-secrets step (Woodpecker v3 requires explicit
# per-secret declaration; there is no expose-all). # per-secret declaration; there is no expose-all).
# #
# ADDING A NEW SECRET (3 small steps, no shell edits): # ADDING A NEW SECRET (3 small steps, no shell edits):
# 1. Add the secret value in Woodpecker UI (repo Settings -> Secrets). # 1. Add the secret value in Woodpecker UI (repo Settings -> Secrets).
@@ -27,7 +27,7 @@
# #
# Stacks not listed here fall through to deploy.yml's legacy case-entries # Stacks not listed here fall through to deploy.yml's legacy case-entries
# untouched. Migration is deliberately one stack per PR. # untouched. Migration is deliberately one stack per PR.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────
stacks: stacks:
ai: ai:
env_template: ai/ai.env.template env_template: ai/ai.env.template
@@ -36,3 +36,6 @@ stacks:
flowagent_azure_client_id: FLOWAGENT_AZURE_CLIENT_ID flowagent_azure_client_id: FLOWAGENT_AZURE_CLIENT_ID
flowagent_azure_tenant_id: FLOWAGENT_AZURE_TENANT_ID flowagent_azure_tenant_id: FLOWAGENT_AZURE_TENANT_ID
flowagent_azure_client_secret: FLOWAGENT_AZURE_CLIENT_SECRET flowagent_azure_client_secret: FLOWAGENT_AZURE_CLIENT_SECRET
traefik:
env_template: traefik/traefik.env.template
env_dest: traefik/traefik.env
+33 -13
View File
@@ -1,5 +1,8 @@
# traefik Stack — Secrets Reference # traefik Stack — Secrets Reference
# Source: traefik.env # Source: traefik.env.template (rendered by deploy/provision-stack.py per
# secrets/secrets-map.yaml — see secrets-map.yaml header for how
# this works). traefik/traefik.env is rendered fresh on every
# provisioning run and is NEVER committed to git.
# #
# Add SECRET values to Woodpecker at: # Add SECRET values to Woodpecker at:
# https://woodpecker.bryanmail.net # https://woodpecker.bryanmail.net
@@ -8,27 +11,44 @@
# ⚠️ HIGH RISK: Traefik is the entry point for all homelab services. # ⚠️ HIGH RISK: Traefik is the entry point for all homelab services.
# If this stack fails, nothing is reachable from outside. # If this stack fails, nothing is reachable from outside.
# Migrate carefully. The Keepalived VIP (192.168.4.30) depends on this stack. # Migrate carefully. The Keepalived VIP (192.168.4.30) depends on this stack.
#
# ⚠️ 2026-09-12 INCIDENT: traefik.env was previously committed to git with a
# literal "***REDACTED***" placeholder as KEEPALIVED_PASSWORD. Every
# git-guard resync/checkout restored that broken value onto disk,
# diverging from keepalived-backup's stale-but-correct in-memory value
# and causing continuous VRRP auth failures + VIP instability. This was
# the trigger for migrating this stack to Pattern C. If you ever see
# KEEPALIVED_PASSWORD as a literal placeholder-looking string on disk
# again, do NOT hand-edit it — check `git log traefik/` for a stray
# commit and fix the template in Gitea instead.
# ── SECRETS (add to Woodpecker) ─────────────────────────────────────────────── # ── SECRETS (add to Woodpecker) ─────────────────────────────────────────
# Woodpecker secret name: traefik_keepalived_password # Woodpecker secret name: traefik_keepalived_password
# Used for: Keepalived VRRP authentication password # Used for: Keepalived VRRP authentication password
# Must match across all 3 nodes (docker-1, docker-2, docker-3) # Must match across all 3 nodes (docker-1, docker-2, docker-3)
# Env var in .env: KEEPALIVED_PASSWORD # NOTE: classic VRRP simple-auth is silently
# truncated to 8 chars by keepalived itself — keep
# the value <= 8 characters, or be aware only the
# first 8 are actually significant on the wire.
# Env var in .env.template: KEEPALIVED_PASSWORD (via TRAEFIK_KEEPALIVED_PASSWORD)
traefik_keepalived_password= traefik_keepalived_password=
# ── NON-SECRETS (safe in compose file or .env) ──────────────────────────────── # ── NON-SECRETS (safe in compose file or .env.template) ─────────────────
# KEEPALIVED_UNICAST_PEERS Python2BASH list of peer IPs # KEEPALIVED_VIRTUAL_IPS Python2BASH list of VIP addresses (192.168.4.30) — literal in template
# KEEPALIVED_VIRTUAL_IPS Python2BASH list of VIP addresses (192.168.4.30)
# ACME_EMAIL Let's Encrypt certificate email # ACME_EMAIL Let's Encrypt certificate email
# TRUSTED_IPS Trusted proxy CIDR ranges # TRUSTED_IPS Trusted proxy CIDR ranges
# TRAEFIK_HOST Traefik dashboard hostname # TRAEFIK_HOST Traefik dashboard hostname
# SPEEDTEST_HOST Speedtest Traefik hostname # SPEEDTEST_HOST Speedtest Traefik hostname
# WHOAMI_HOST Whoami Traefik hostname # WHOAMI_HOST Whoami Traefik hostname
# ── Woodpecker provision-secrets case entry ─────────────────────────────────── # ── Provisioning (manifest-driven, deploy/provision-stack.py) ───────────
# #
# traefik) # This stack is migrated — provisioning happens automatically via:
# create_or_update_secret "traefik_keepalived_password" "$TRAEFIK_KEEPALIVED_PASSWORD" # secrets/secrets-map.yaml (traefik: entry)
# ;; # traefik/traefik.env.template (authoritative key list)
# deploy/provision-stack.py (renders + ships traefik/traefik.env)
#
# The .woodpecker/deploy.yml provision-secrets step calls this with a
# single line: `python3 deploy/provision-stack.py traefik`
-2
View File
@@ -1,2 +0,0 @@
KEEPALIVED_PASSWORD=***REDACTED***
KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['192.168.4.30']"
+52
View File
@@ -0,0 +1,52 @@
# ─────────────────────────────────────────────────────────────────────────
# traefik.env.template — AUTHORITATIVE template for traefik/traefik.env
# (rendered by deploy/provision-stack.py per secrets/secrets-map.yaml)
#
# - This file IS the complete key list for traefik.env. The whole file is
# rendered on every provisioning run — no line surgery, so a key can
# never silently go missing.
# - Non-secret config lives here as LITERAL values (visible, reviewable).
# - Secret values are dollar-brace placeholders resolved from the CI env
# (Woodpecker from_secret vars) at provisioning time. provision-stack.py
# FAILS HARD if any placeholder is missing/empty.
# - The rendered traefik/traefik.env exists only on the host (gitignored).
# - Rendered by provision-stack.py, NOT Woodpecker's yaml preprocessor —
# single-dollar placeholders are safe here (deploy.yml's double-dollar
# rule does NOT apply to this file).
#
# Consumed by traefik/traefik.yaml (KEEPALIVED_PASSWORD, KEEPALIVED_VIRTUAL_IPS).
# DOMAIN comes from deploy/global.env, not here.
#
# ⚠️ HIGH RISK: Traefik is the entry point for all homelab services, and the
# Keepalived VIP (192.168.4.30) depends on this stack. KEEPALIVED_PASSWORD
# MUST be identical across keepalived-master and keepalived-backup — both
# consume this same rendered value — or VRRP auth fails and the VIP
# becomes unstable (this is exactly what the 2026-09-12 incident was:
# a committed literal "***REDACTED***" placeholder in git, restored every
# time the file was deleted/re-synced, diverging from -backup's stale but
# correct in-memory value).
#
# Classic VRRP simple-auth is silently truncated to 8 characters by
# keepalived itself. Set TRAEFIK_KEEPALIVED_PASSWORD in Woodpecker to a
# value 8 characters or fewer (or accept that only the first 8 chars are
# actually significant) so the effective negotiated value is unambiguous.
#
# 2026-09-12 RE-TRIGGER: traefik.env was found MISSING on disk after the
# Pattern C migration merged — the manifest-driven provisioning render only
# happens on a push that changes traefik/, and traefik is bootstrap-tier
# (stack-deploy.sh must be run manually; the pipeline never auto-deploys
# this stack). A manual `stack-deploy.sh traefik` had apparently been run
# while traefik.env was absent, causing Swarm's own interpolation to
# silently render KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS as EMPTY
# STRINGS on keepalived-master (not missing — empty, which is worse,
# since it's a valid-looking VRRP auth string that mismatches everything).
# This comment-only change re-triggers provision-secrets to re-render and
# re-ship traefik.env; a manual `stack-deploy.sh traefik` must still be run
# afterward to actually apply it to the running services.
# ─────────────────────────────────────────────────────────────────────────
# ── Keepalived (non-secret config) ────────────────────────────────────────
KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['192.168.4.30']"
# ── Keepalived (secret) ────────────────────────────────────────────────────
KEEPALIVED_PASSWORD=${TRAEFIK_KEEPALIVED_PASSWORD}