Author SHA1 Message Date
Bot 5ab1ae1e8b chore(traefik): re-trigger provision-secrets — traefik.env was found missing on disk post-migration
ci/woodpecker/push/deploy Pipeline was successful
traefik/traefik.env did not exist on the host after PR #21 merged. Root
cause: provisioning only renders on a push touching traefik/, and traefik
is bootstrap-tier (stack-deploy.sh runs manually, never via the pipeline).
A manual stack-deploy.sh traefik run apparently happened while the env file
was absent, causing Swarm's own interpolation to render
KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS as empty strings on
keepalived-master. This comment-only change re-triggers provision-secrets;
a manual stack-deploy.sh traefik run is still required afterward to apply
the freshly-rendered env to the running services.
2026-09-11 23:17:13 -07:00
AVB afc96a6d37 Merge pull request 'fix(traefik): migrate KEEPALIVED_PASSWORD to Pattern C (manifest-driven), stop committing traefik.env' (#21) from migrate-traefik-keepalived-secret into main
ci/woodpecker/push/deploy Pipeline was successful
Reviewed-on: #21
https://ai.bryanmail.net/c/7440e9d5-7e39-40f9-a35a-33016b6ae8d1
2026-09-11 22:52:35 -07:00
Bot cd42e35265 docs(secrets): mark traefik migration complete, add committed-env-file warning 2026-09-11 22:42:06 -07:00
Bot 2a3f2bdd1e docs(traefik): update secrets reference for manifest-driven Pattern C migration 2026-09-11 22:42:04 -07:00
Bot d9f151a1c3 feat(traefik): migrate keepalived password to manifest-driven Pattern C provisioning
Root cause of the 2026-09-12 VIP instability incident: traefik/traefik.env
was committed to git with a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD. Every git-guard resync/checkout restored that broken
value onto disk, diverging from keepalived-backup's stale-but-correct
in-memory value and causing a continuous VRRP auth failure + VIP flapping.

- Adds TRAEFIK_KEEPALIVED_PASSWORD (from_secret: traefik_keepalived_password)
  to provision-secrets environment block
- Replaces the traefik legacy case-entry (previously grouped with
  maintenance|media|unifi|... as "no Docker secrets, host .env only") with
  a call to deploy/provision-stack.py traefik, matching the ai) stack's
  manifest-driven pattern
- traefik/traefik.env is now rendered fresh from traefik/traefik.env.template
  on every provisioning run and never committed to git again

Diffed $${ occurrence count against previous version before committing,
per this file's own 2026-08-26 incident note.
2026-09-11 22:40:36 -07:00
Bot 201d8418ee feat(traefik): register traefik in manifest-driven secrets provisioning 2026-09-11 22:38:39 -07:00
Bot 5edebd977d fix(traefik): remove tracked traefik.env — never commit secret-bearing env files
The committed version contained a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD, which was restored onto disk every time the real file
was deleted or the local checkout re-synced from git. This caused VRRP auth
to mismatch between keepalived-master and keepalived-backup (master picked
up the broken committed value, backup retained a stale-but-correct value),
producing a continuous "(VI_1) received an invalid passwd!" flood and VIP
instability. traefik.env is now rendered fresh from traefik.env.template by
deploy/provision-stack.py on every provisioning run and never committed.
2026-09-11 22:38:39 -07:00
Bot ffec3ea5b7 feat(traefik): add authoritative env template for manifest-driven provisioning 2026-09-11 22:38:38 -07:00
Bot 5dda5e0031 fix(frigate): publish go2rtc WebRTC port 8555 (tcp+udp) on frigate-nvr
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
Camera cards using advanced-camera-card were stuck retrying WebRTC ICE
negotiation every few seconds (visible as dashboards "refreshing").
frigate-nvr had no published ports at all, so go2rtc's WebRTC listener
on 8555 was unreachable from browsers despite valid ICE candidates
already configured (VIP + CIDR + stun) in frigate/config.yml.

Publishing via ingress mode keeps this working regardless of which
swarm node Frigate lands on (no placement constraint on this service).

No Traefik/auth changes - HA<->Frigate signaling was already working
correctly over traefik_backend; only the browser<->go2rtc media path
was broken.
2026-09-09 22:09:30 -07:00
AVB d5db4c1a4e fix(git-guard): use 'exec bash "$0"' for self-re-invocation (exec bit not required) (#20)
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
2026-09-09 19:36:31 -07:00
Bot 333bb82c16 fix(git-guard): use 'exec bash "$0"' instead of 'exec "$0"' for self-re-invocation
Found via live testing on docker-2 after merging #19: git-guard.sh is
tracked in git at mode 100644 (not executable). stack-deploy.sh correctly
invokes it as `bash deploy/git-guard.sh`, sidestepping the exec bit for the
first call -- but the script's own internal `exec "$0" "$@"` calls (used to
cleanly re-run after a successful push or stash-pop) rely on the kernel
executing the file directly, which requires +x. Since every fresh checkout
or fast-forward preserves the tracked 644 mode, this failed with
"Permission denied" (exit 126) the moment either self-re-invocation path
was actually exercised.

Confirmed via a live dry run: simulated the exact #18 incident shape (dirty
git-guard.sh on a HEAD 3 commits behind origin/main) on docker-2. The new
stash-first logic from #19 worked perfectly end-to-end -- detected
dirty+stale, stashed safely, fast-forwarded via resync_with_origin(), and
popped the stash cleanly -- but then hit this pre-existing bug on the
final `exec "$0" "$@"` re-invocation. This bug pre-dates #19 (the old
dirty-commit-then-push-success path had the identical pattern); #19 just
added a second trigger point that happened to surface it during testing.

Fix: `exec bash "$0" "$@"` explicitly invokes through the interpreter
instead of relying on the file's own execute bit -- correct regardless of
what git tracks the file's mode as.
2026-09-09 14:01:26 -07:00
Bot d72a8ebd04 Merge pull request 'fix(git-guard): stash-before-commit when dirty tree is also stale vs origin' (#19) from fix-git-guard-stash-before-commit into main
ci/woodpecker/push/deploy Pipeline was successful
2026-09-09 13:56:05 -07:00
Bot e357907ee6 fix(git-guard): add unmerged-path guard + concrete remediation syntax for every failure branch
Adds the unmerged-conflict guard flagged in PR review: before the dirty-tree
flow can run, check `git ls-files -u` and refuse to auto-commit if unresolved
merge conflict markers are present (e.g. left behind by a prior run's failed
`git stash pop`). Without this, a re-run's `git add -A` would silently stage
literal <<<<<<< / ======= / >>>>>>> markers into a real commit and push them
to origin/main.

Also expands every failure-path message (stash-push failure, stash-pop
conflict vs. untracked-file-collision, resync failure, true divergence,
manual-abort cases) to include concrete, copy-pasteable remediation command
sequences with multiple options (resolve-in-place vs. abandon-and-reapply vs.
drop-if-unneeded), rather than a single generic hint. The true-divergence
message now also tells the operator to check `git stash list` first, so a
stash created by the new dirty+stale path isn't confused with a second
manually-created one.

No change to the common dirty-but-current or clean-but-stale behavior.
2026-09-09 13:47:14 -07:00
Bot bd69cc85d5 fix(git-guard): stash-before-commit when dirty tree is also stale vs origin
Previously the dirty-tree branch committed unconditionally, then only
discovered staleness/divergence when the push was rejected -- leaving a
doomed auto-commit sitting on a stale base. Root-caused from a real
incident: a stray on-disk edit to ai/ai.yaml sat on a checkout that was
already 4 commits behind, git-guard auto-committed on top of it, then
the push was rejected (ahead 1, behind 4), aborting the deploy.

Fix: when DIRTY=1 and LOCAL != REMOTE, stash the dirty changes first,
resync main with origin using the exact same behind/ahead/diverged
rules as the clean-tree path (now shared via resync_with_origin()),
then reapply the stash and re-run. Never commits on top of a stale
base again. On any failure the stash is preserved and printed for
manual recovery -- changes are never silently lost.
2026-09-09 10:42:21 -07:00
AVB 000792f702 ai: persist FlowAgent MSAL token cache via mcpo bind mount (#18)
ci/woodpecker/push/deploy Pipeline was successful
https://ai.bryanmail.net/s/0255f5cf-cb2a-4f8d-a782-caacc9dfd62f
2026-09-09 08:45:02 -07:00
Bot 9316a32ba5 ai: persist FlowAgent MSAL token cache via mcpo bind mount
Adds the missing bind mount for FlowAgent's delegated-user MSAL disk
token cache:

    /volume1/docker/mcpo/flowagent-auth:/app/flowagent-auth

Confirmed via `docker service inspect ai_mcpo --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'`-equivalent review of the current
compose that mcpo's mounts were only: ssh_keys, config.json, mcpo_data,
uv-cache -- flowagent-auth (set as HOME for the flowagent subprocess via
config.json.template's env.HOME=/app/flowagent-auth) was NOT persisted,
so any successful delegated-user MSAL sign-in (list_connections /
create_connection / resolve_refs / pick_or_create_connection /
resolve_params -- the tools that use MsalTokenProvider, see companion
homelab/mcp-config PR #9) would be silently lost on every mcpo restart,
forcing a fresh interactive sign-in every time.

Host dir already exists (pre-created on shared CephFS, identical across
all three docker LXCs):
    mkdir -p /volume1/docker/mcpo/flowagent-auth

This does NOT change the flowagent image tag/pin -- that stays
git.bryanmail.net/homelab/flowagent-mcpo:c5b22618404a pending the
companion mcp-config PR merging and its Woodpecker build producing a new
commit-sha tag to bump to.

No other changes to ai.yaml in this commit -- diff is mount-list addition
only.
2026-09-09 00:16:35 -07:00
AVB 5a5c8e56a9 Merge pull request 'Fix phantom verify failure: secrets/ is not a stack + errexit-proof verify' (#17) from fix-secrets-folder-stack-detection into main
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline failed
Reviewed-on: #17
https://ai.bryanmail.net/c/9f4c84c7-7aa1-4b24-a27e-98be6a5e737e
2026-09-07 22:28:36 -07:00
9 changed files with 415 additions and 103 deletions
+27 -1
View File
@@ -70,6 +70,16 @@ when:
# tolerating a failing stack-ps in verify (|| true) so a genuinely missing
# stack produces the designed WARNING instead of killing the step. This
# hazard was first flagged in July (PR #3, closed unmerged).
#
# 2026-09-12 MIGRATION: traefik moved from Pattern B (secrets hand-typed
# into the host-only traefik.env, no CI involvement at all) to the same
# manifest-driven provisioning as ai) below. Root cause of the migration:
# traefik.env had been accidentally committed to git with a literal
# "***REDACTED***" placeholder as KEEPALIVED_PASSWORD; every git-guard
# resync/checkout silently restored that broken value, which diverged from
# keepalived-backup's stale-but-correct in-memory value and produced a
# continuous VRRP auth failure ("received an invalid passwd!") and VIP
# instability. The manifest-driven path never commits the rendered file.
# ─────────────────────────────────────────────────────────────────────────────
steps:
@@ -216,6 +226,11 @@ steps:
from_secret: flowagent_azure_tenant_id
FLOWAGENT_AZURE_CLIENT_SECRET:
from_secret: flowagent_azure_client_secret
# ── traefik stack (manifest-driven — secrets/secrets-map.yaml +
# traefik/traefik.env.template + deploy/provision-stack.py).
# Migrated 2026-09-12; see header note above for root cause. ──
TRAEFIK_KEEPALIVED_PASSWORD:
from_secret: traefik_keepalived_password
commands:
- apk add --no-cache openssh-client python3 py3-yaml
- mkdir -p ~/.ssh
@@ -243,8 +258,19 @@ steps:
for STACK in $ALL_STACKS; do
echo "Provisioning: $STACK"
case "$STACK" in
maintenance|media|unifi|guacamole|security|auth|traefik|meshcentral|ddm)
maintenance|media|unifi|guacamole|security|auth|meshcentral|ddm)
echo " No Docker secrets for $STACK — secrets in host .env";;
traefik)
# MIGRATED (2026-09-12) to manifest-driven provisioning after
# discovering traefik.env was committed to git with a literal
# "***REDACTED***" placeholder as KEEPALIVED_PASSWORD — every
# git-guard resync/checkout restored the broken value, causing
# VRRP auth mismatch between keepalived-master/-backup and VIP
# instability. All logic lives in deploy/provision-stack.py;
# the authoritative key list lives in
# traefik/traefik.env.template; the mapping lives in
# secrets/secrets-map.yaml. This case is intentionally one line.
python3 deploy/provision-stack.py traefik;;
immich)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'immich_db_password' '$${IMMICH_DB_PASSWORD}'
+1
View File
@@ -141,6 +141,7 @@ services:
- /volume1/docker/mcpo/data:/mcpo_data
- /volume1/docker/cronicle/ssh_keys:/app/ssh_keys:ro
- /volume1/docker/mcpo/uv-cache:/app/uv-cache
- /volume1/docker/mcpo/flowagent-auth:/app/flowagent-auth
secrets:
- source: flowagent_azure_client_id
target: flowagent_azure_client_id
+258 -68
View File
@@ -5,10 +5,20 @@
# Behavior:
# - Clean + up to date -> pass silently
# - Clean + behind (ff-only) -> auto `git pull --ff-only`, then pass
# - Ahead only (unpushed) -> interactive: offer to push; non-interactive: BLOCK
# - Dirty tracked changes -> offer to commit + push right now
# (auto in non-interactive/CI runs, after a
# - Clean + ahead only -> interactive: offer to push; non-interactive: BLOCK
# (unpushed)
# - Unresolved merge conflict -> REFUSE immediately. Never auto-commits over
# markers present conflict markers. Prints remediation options.
# - Dirty + local in sync -> offer to commit + push right now
# with origin (auto in non-interactive/CI runs, after a
# secret-pattern scan of the staged diff)
# - Dirty + local STALE/ -> NEVER commit on top of a stale base. Stash
# diverged vs origin the dirty changes first, resync main with
# origin using the same behind/ahead/diverged
# rules as the clean-tree case, then reapply
# the stash and re-run. On any failure the
# stash is preserved and remediation options
# (with exact commands) are printed.
# - Diverged (local AND -> REFUSE. Never auto-resolves. Prints the
# remote both moved) backup/stash/reset recovery steps and exits.
#
@@ -36,12 +46,246 @@ git status --porcelain | grep -q . && DIRTY=1
SECRET_PATTERN='(-----BEGIN [A-Z]+ PRIVATE KEY-----|AKIA[0-9A-Z]{16}|xox[baprs]-[0-9a-zA-Z-]+|password[[:space:]]*[:=][[:space:]]*[^$ ]|api[_-]?key[[:space:]]*[:=][[:space:]]*[^$ ])'
# resync_with_origin <local_sha> <remote_sha> <base_sha>
#
# Handles the behind/ahead/diverged cases against a CLEAN working tree.
# Shared by both the "tree was already clean" path and the new
# "dirty tree turned out to be stale, so we stashed first" path, so the
# two paths can never drift out of sync with each other.
#
# Returns 0 if it's now safe to deploy, 1 if it could not safely resolve
# (guidance already printed to stdout in that case).
resync_with_origin() {
local local_sha="$1" remote_sha="$2" base_sha="$3"
# ---- Case: fully in sync ----
if [ "$local_sha" = "$remote_sha" ]; then
echo "==> In sync with origin/main ($local_sha). OK to deploy."
return 0
fi
# ---- Case: behind only (fast-forwardable) ----
if [ "$local_sha" = "$base_sha" ]; then
echo "!! Local main is behind origin/main."
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Fast-forward pull now? [y/N] " ans
else
ans="y"
echo "(non-interactive session — auto fast-forwarding)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git pull --ff-only origin main
echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
return 0
else
echo "Aborting - pull manually, then retry:"
echo " cd $DIR && git pull --ff-only origin main"
return 1
fi
fi
# ---- Case: ahead only (local commits not yet pushed) ----
if [ "$remote_sha" = "$base_sha" ]; then
echo "!! Local main is AHEAD of origin/main (unpushed commits):"
git log --oneline "origin/main..main"
echo
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Push local commits to origin/main now? [y/N] " ans
else
ans="n"
echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git push origin main
echo "==> Pushed. OK to deploy."
return 0
else
echo "Aborting. Review with:"
echo " cd $DIR && git log origin/main..main"
echo "Then push manually when ready:"
echo " git push origin main"
return 1
fi
fi
# ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!! DIVERGED: local main and origin/main have both moved independently."
echo "!!"
echo "!! Local-only commits:"
git log --oneline "$base_sha..main" | sed 's/^/!! /'
echo "!!"
echo "!! Remote-only commits:"
git log --oneline "$base_sha..origin/main" | sed 's/^/!! /'
echo "!!"
echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
echo "!!"
echo "!! Before stashing anything new, check whether a git-guard safety stash"
echo "!! ALREADY exists from this same run (avoids confusing duplicate stashes):"
echo "!! git stash list"
echo "!!"
echo "!! Recommended recovery:"
echo "!! 1. tar backup: tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
echo "!! 2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
echo "!! 3. stash any NEW uncommitted state only if 'git stash list' above"
echo "!! didn't already show one for this run:"
echo "!! git stash push -u -m 'pre-reset-snapshot'"
echo "!! 4. reset to origin: git reset --hard origin/main"
echo "!! 5. selectively restore needed files from the stash/backup branch:"
echo "!! git stash list"
echo "!! git stash show -p stash@{N}"
echo "!! git stash apply stash@{N} # 'apply' keeps the stash as a backup; use 'pop' to also drop it"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
return 1
}
# ---- Case: unresolved merge conflict already present ----
# Can happen if a PRIOR git-guard run's `git stash pop` conflicted and the
# resulting conflict markers were never resolved before the next deploy
# attempt. Must be checked BEFORE the dirty-tree commit flow below, because
# an unmerged path shows up as "dirty" too, and `git add -A` would silently
# stage the literal <<<<<<< / ======= / >>>>>>> markers into a real commit.
if git ls-files -u | grep -q .; then
echo "ERROR: unresolved merge conflict markers present in the working tree."
echo "Refusing to auto-commit over a conflict — this would push literal"
echo "<<<<<<< / ======= / >>>>>>> markers to origin/main."
echo
echo "Conflicted paths:"
git diff --name-only --diff-filter=U | sed 's/^/ /'
echo
echo "Remediation options:"
echo " A) Resolve the conflict by hand, then commit and push:"
echo " cd $DIR"
echo " git status # see conflicted paths"
echo " git diff # inspect the conflict markers"
echo " \$EDITOR <conflicted-file> # remove markers, keep correct content"
echo " git add <conflicted-file>"
echo " git commit -m 'resolve git-guard stash-pop conflict'"
echo " git push origin main"
echo " B) Discard the conflicted merge attempt entirely and start clean from"
echo " origin/main, then decide separately whether to re-apply anything"
echo " from a prior safety stash:"
echo " cd $DIR"
echo " git checkout -- ."
echo " git reset --hard origin/main"
echo " git stash list # look for a git-guard-safety-stash-* entry"
echo " git stash show -p stash@{N} # inspect before deciding"
echo " C) Once resolved (via A or B) and no longer needed, clean up the stash:"
echo " git stash drop stash@{N}"
exit 1
fi
# ---- Case: dirty tracked changes ----
if [ "$DIRTY" -eq 1 ]; then
echo "!! WORKING TREE DIRTY — uncommitted changes detected:"
git status --short
echo
# If local is ALSO stale/diverged from origin, committing right now would
# create a doomed commit on top of a base that's about to be rejected on
# push (this is exactly what caused a real incident: a stray on-disk edit
# sat on a checkout that was 4 commits behind, git-guard auto-committed
# anyway, then the push bounced). Stash first, resync safely using the
# same rules as the clean-tree path, then reapply.
if [ "$LOCAL" != "$REMOTE" ]; then
echo "!! Local main is ALSO stale/diverged from origin/main."
echo " Refusing to commit on top of a stale base — stashing the dirty"
echo " changes safely first, then resyncing with origin."
echo
STASH_MSG="git-guard-safety-stash-$(date -u +%Y%m%dT%H%M%SZ)"
if ! git stash push -u -m "$STASH_MSG"; then
echo "ERROR: 'git stash push' itself failed (disk full, permissions, or"
echo " some other git error). Your changes are still on disk,"
echo " uncommitted — nothing has been lost, but git-guard cannot"
echo " proceed safely until this is resolved."
echo
echo "Remediation options:"
echo " A) Check disk space and permissions, then retry the deploy:"
echo " df -h $DIR"
echo " ls -la $DIR"
echo " B) Identify and manually move aside whatever is blocking the stash,"
echo " then retry:"
echo " cd $DIR"
echo " git status --short # find the offending path(s)"
echo " mv <path> <path>.bak-\$(date +%s)"
echo " C) Inspect the raw git error above for specifics before proceeding."
exit 1
fi
echo "==> Stashed as: $STASH_MSG"
if resync_with_origin "$LOCAL" "$REMOTE" "$BASE"; then
echo "==> Resync succeeded. Reapplying stashed changes..."
if git stash pop; then
echo "==> Stash reapplied cleanly. Re-checking sync state..."
exec bash "$0" "$@"
else
echo "ERROR: 'git stash pop' did not complete successfully."
echo
if git ls-files -u | grep -q .; then
echo "This is a MERGE CONFLICT — your stashed changes were partially"
echo "applied and conflict markers (<<<<<<< / ======= / >>>>>>>) are now"
echo "in the working tree. The stash itself is still preserved as a backup."
echo
echo "Remediation options:"
echo " A) Resolve the conflict by hand, then commit and push:"
echo " cd $DIR"
echo " git status # see conflicted paths"
echo " git diff # inspect the markers"
echo " \$EDITOR <conflicted-file> # remove markers, keep correct content"
echo " git add <conflicted-file>"
echo " git commit -m 'resolve git-guard stash-pop conflict'"
echo " git push origin main"
echo " git stash list # confirm which entry is: $STASH_MSG"
echo " git stash drop stash@{N} # once confirmed no longer needed"
echo " B) Abandon this merge attempt and fall back to a clean, resynced"
echo " tree, then re-apply the change manually with full visibility:"
echo " cd $DIR"
echo " git checkout -- ."
echo " git reset --hard origin/main # now matches origin, no conflict"
echo " git stash list # find: $STASH_MSG"
echo " git stash show -p stash@{N} # review the content"
echo " git stash apply stash@{N} # 'apply' keeps the backup; use 'pop' to also drop it"
else
echo "This looks like an UNTRACKED-FILE COLLISION, not a merge conflict"
echo "(a file added upstream shares a path with an untracked file in your"
echo "stash). No conflict markers were written; the stash was NOT applied"
echo "and remains fully intact."
echo
echo "Remediation options:"
echo " A) Move the colliding upstream file aside, pop, then reconcile:"
echo " cd $DIR"
echo " git status --short # identify the colliding path"
echo " mv <path> <path>.upstream-\$(date +%s)"
echo " git stash pop"
echo " diff <path> <path>.upstream-* # reconcile manually, then remove the .upstream-* backup"
echo " B) Inspect the stash without applying, and hand-merge the needed"
echo " pieces instead:"
echo " git stash list # find N"
echo " git stash show -p stash@{N}"
fi
echo
echo "Your stash reference for this run: $STASH_MSG"
exit 1
fi
else
echo "ERROR: could not safely resync with origin/main."
echo "Your uncommitted changes are preserved in the stash: $STASH_MSG"
echo
echo "Remediation options:"
echo " A) Follow the manual recovery steps printed above (from the"
echo " behind/ahead/diverged case), THEN reapply your change:"
echo " cd $DIR"
echo " git stash list # find: $STASH_MSG"
echo " git stash apply stash@{N} # or 'pop' to also drop it once resynced"
echo " B) If the stashed change is no longer needed (e.g. it's already"
echo " represented in a since-merged PR), verify then drop it:"
echo " git stash show -p stash@{N}"
echo " git stash drop stash@{N}"
exit 1
fi
fi
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Commit and push these changes to origin/main now? [y/N] " ans
else
@@ -54,7 +298,8 @@ if [ "$DIRTY" -eq 1 ]; then
if git diff --cached | grep -Eiq "$SECRET_PATTERN"; then
echo "ERROR: possible secret detected in staged changes. Refusing to auto-commit."
echo "Review manually: git diff --cached"
echo "Review manually:"
echo " cd $DIR && git diff --cached"
git reset
exit 1
fi
@@ -63,80 +308,25 @@ if [ "$DIRTY" -eq 1 ]; then
if git push origin main; then
echo "==> Pushed. Re-checking sync state..."
exec "$0" "$@"
exec bash "$0" "$@"
else
echo "ERROR: push failed (likely diverged from origin). Aborting deploy."
echo "Run: cd $DIR && git status"
echo "Run:"
echo " cd $DIR && git status"
exit 1
fi
else
echo "Aborting deploy - commit or stash changes manually, then retry."
echo "Aborting deploy - commit or stash changes manually, then retry:"
echo " cd $DIR"
echo " git add -A && git commit -m 'your message' && git push origin main"
echo " # or: git stash push -u -m 'manual-stash'"
exit 1
fi
fi
# ---- Case: fully in sync ----
if [ "$LOCAL" = "$REMOTE" ]; then
echo "==> In sync with origin/main ($LOCAL). OK to deploy."
exit 0
fi
# ---- Case: behind only (fast-forwardable) ----
if [ "$LOCAL" = "$BASE" ]; then
echo "!! Local main is behind origin/main."
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Fast-forward pull now? [y/N] " ans
else
ans="y"
echo "(non-interactive session — auto fast-forwarding)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git pull --ff-only origin main
echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
# ---- Clean tree: resync with origin using the shared logic above ----
if resync_with_origin "$LOCAL" "$REMOTE" "$BASE"; then
exit 0
else
echo "Aborting deploy - pull manually, then retry."
exit 1
fi
fi
# ---- Case: ahead only (local commits not yet pushed) ----
if [ "$REMOTE" = "$BASE" ]; then
echo "!! Local main is AHEAD of origin/main (unpushed commits):"
git log --oneline "origin/main..main"
echo
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Push local commits to origin/main now? [y/N] " ans
else
ans="n"
echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git push origin main
echo "==> Pushed. OK to deploy."
exit 0
else
echo "Aborting deploy. Review with: git log origin/main..main"
exit 1
fi
fi
# ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!! DIVERGED: local main and origin/main have both moved independently."
echo "!!"
echo "!! Local-only commits:"
git log --oneline "$BASE..main" | sed 's/^/!! /'
echo "!!"
echo "!! Remote-only commits:"
git log --oneline "$BASE..origin/main" | sed 's/^/!! /'
echo "!!"
echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
echo "!! Recommended recovery:"
echo "!! 1. tar backup: tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
echo "!! 2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
echo "!! 3. stash all state: git stash push -u -m 'pre-reset-snapshot'"
echo "!! 4. reset to origin: git reset --hard origin/main"
echo "!! 5. selectively restore needed files from the stash/backup branch"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
exit 1
+9
View File
@@ -63,6 +63,15 @@ services:
FRIGATE_RTSP_PASSWORD: "${FRIGATE_RTSP_PASSWORD}"
networks:
- traefik_backend
ports:
- target: 8555
published: 8555
protocol: udp
mode: ingress
- target: 8555
published: 8555
protocol: tcp
mode: ingress
deploy:
labels:
- traefik.enable=true
+18 -5
View File
@@ -6,9 +6,9 @@ This directory documents the secrets required for each Docker Swarm stack.
```
Secret values live in Woodpecker (encrypted)
pipeline reads them at deploy time
pipeline reads them at deploy time
Docker Swarm secret store (encrypted Raft DB, replicated across all nodes)
mounted into containers at runtime
mounted into containers at runtime
/run/secrets/<secret_name>
```
@@ -38,7 +38,10 @@ Use the stack's `.secrets.example` file as your checklist.
### Step 2 — Add the stack's case to `.woodpecker.yml`
In the `provision-secrets` step, add a case for the stack that calls
`create_or_update_secret` for each secret.
`create_or_update_secret` for each secret. (Or, preferred for new
migrations: add an entry to `secrets/secrets-map.yaml` + a
`<stack>.env.template` and call `deploy/provision-stack.py <stack>`
instead — see the `ai` and `traefik` entries for the current pattern.)
### Step 3 — Test by pushing a trivial change to the stack's yaml file
Watch the pipeline run: provision-secrets → validate → deploy → verify → notify.
@@ -71,10 +74,20 @@ Examples:
- **PostgreSQL is highest risk.** Its master password is used by nearly every
other stack. Migrate it last.
- **Never commit a rendered env file, even by accident.** `traefik/traefik.env`
was committed to git for a period (discovered/fixed 2026-09-12) with a
literal "***REDACTED***" placeholder as KEEPALIVED_PASSWORD, which was
silently restored every time the file was deleted or the checkout resynced
from git — causing a real VRRP auth outage. `.gitignore` blanket-excludes
`*.env`, but that rule does NOT retroactively untrack a file already
committed before the rule existed. If you ever see a stack's `.env` file
show up in `git status` as tracked, stop and untrack it (`git rm --cached`)
before doing anything else.
## Migration Status
| Stack | Secrets in Woodpecker | Pipeline Step Added | .env Removed |
|-------|----------------------|---------------------|--------------|
|-------|------------------------|----------------------|---------------|
| 3dprint | ⏳ | ⏳ | ⏳ |
| ai | ⏳ | ⏳ | ⏳ |
| auth | ⏳ | ⏳ | ⏳ |
@@ -91,7 +104,7 @@ Examples:
| postgresql | ⏳ | ⏳ | ⏳ |
| productivity | ⏳ | ⏳ | ⏳ |
| security | ⏳ | ⏳ | ⏳ |
| traefik | ⏳ | ⏳ | ⏳ |
| traefik | ✅ (manifest-driven, 2026-09-12) | ✅ (manifest-driven, 2026-09-12) | ⏳ |
| unifi | ✅ N/A (no secrets) | ✅ N/A | ⏳ |
| vaultwarden | ⏳ | ⏳ | ⏳ |
| woodpecker | ⏳ Manual only | ⏳ N/A | ⏳ |
+5 -2
View File
@@ -1,4 +1,4 @@
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
# secrets-map.yaml — DATA-ONLY manifest for deploy/provision-stack.py
#
# RULES:
@@ -27,7 +27,7 @@
#
# Stacks not listed here fall through to deploy.yml's legacy case-entries
# untouched. Migration is deliberately one stack per PR.
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
stacks:
ai:
env_template: ai/ai.env.template
@@ -36,3 +36,6 @@ stacks:
flowagent_azure_client_id: FLOWAGENT_AZURE_CLIENT_ID
flowagent_azure_tenant_id: FLOWAGENT_AZURE_TENANT_ID
flowagent_azure_client_secret: FLOWAGENT_AZURE_CLIENT_SECRET
traefik:
env_template: traefik/traefik.env.template
env_dest: traefik/traefik.env
+30 -10
View File
@@ -1,5 +1,8 @@
# traefik Stack — Secrets Reference
# Source: traefik.env
# Source: traefik.env.template (rendered by deploy/provision-stack.py per
# secrets/secrets-map.yaml — see secrets-map.yaml header for how
# this works). traefik/traefik.env is rendered fresh on every
# provisioning run and is NEVER committed to git.
#
# Add SECRET values to Woodpecker at:
# https://woodpecker.bryanmail.net
@@ -8,27 +11,44 @@
# ⚠️ HIGH RISK: Traefik is the entry point for all homelab services.
# If this stack fails, nothing is reachable from outside.
# Migrate carefully. The Keepalived VIP (192.168.4.30) depends on this stack.
#
# ⚠️ 2026-09-12 INCIDENT: traefik.env was previously committed to git with a
# literal "***REDACTED***" placeholder as KEEPALIVED_PASSWORD. Every
# git-guard resync/checkout restored that broken value onto disk,
# diverging from keepalived-backup's stale-but-correct in-memory value
# and causing continuous VRRP auth failures + VIP instability. This was
# the trigger for migrating this stack to Pattern C. If you ever see
# KEEPALIVED_PASSWORD as a literal placeholder-looking string on disk
# again, do NOT hand-edit it — check `git log traefik/` for a stray
# commit and fix the template in Gitea instead.
# ── SECRETS (add to Woodpecker) ───────────────────────────────────────────────
# ── SECRETS (add to Woodpecker) ─────────────────────────────────────────
# Woodpecker secret name: traefik_keepalived_password
# Used for: Keepalived VRRP authentication password
# Must match across all 3 nodes (docker-1, docker-2, docker-3)
# Env var in .env: KEEPALIVED_PASSWORD
# NOTE: classic VRRP simple-auth is silently
# truncated to 8 chars by keepalived itself — keep
# the value <= 8 characters, or be aware only the
# first 8 are actually significant on the wire.
# Env var in .env.template: KEEPALIVED_PASSWORD (via TRAEFIK_KEEPALIVED_PASSWORD)
traefik_keepalived_password=
# ── NON-SECRETS (safe in compose file or .env) ────────────────────────────────
# ── NON-SECRETS (safe in compose file or .env.template) ─────────────────
# KEEPALIVED_UNICAST_PEERS Python2BASH list of peer IPs
# KEEPALIVED_VIRTUAL_IPS Python2BASH list of VIP addresses (192.168.4.30)
# KEEPALIVED_VIRTUAL_IPS Python2BASH list of VIP addresses (192.168.4.30) — literal in template
# ACME_EMAIL Let's Encrypt certificate email
# TRUSTED_IPS Trusted proxy CIDR ranges
# TRAEFIK_HOST Traefik dashboard hostname
# SPEEDTEST_HOST Speedtest Traefik hostname
# WHOAMI_HOST Whoami Traefik hostname
# ── Woodpecker provision-secrets case entry ───────────────────────────────────
# ── Provisioning (manifest-driven, deploy/provision-stack.py) ───────────
#
# traefik)
# create_or_update_secret "traefik_keepalived_password" "$TRAEFIK_KEEPALIVED_PASSWORD"
# ;;
# This stack is migrated — provisioning happens automatically via:
# secrets/secrets-map.yaml (traefik: entry)
# traefik/traefik.env.template (authoritative key list)
# deploy/provision-stack.py (renders + ships traefik/traefik.env)
#
# The .woodpecker/deploy.yml provision-secrets step calls this with a
# single line: `python3 deploy/provision-stack.py traefik`
-2
View File
@@ -1,2 +0,0 @@
KEEPALIVED_PASSWORD=***REDACTED***
KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['192.168.4.30']"
+52
View File
@@ -0,0 +1,52 @@
# ─────────────────────────────────────────────────────────────────────────
# traefik.env.template — AUTHORITATIVE template for traefik/traefik.env
# (rendered by deploy/provision-stack.py per secrets/secrets-map.yaml)
#
# - This file IS the complete key list for traefik.env. The whole file is
# rendered on every provisioning run — no line surgery, so a key can
# never silently go missing.
# - Non-secret config lives here as LITERAL values (visible, reviewable).
# - Secret values are dollar-brace placeholders resolved from the CI env
# (Woodpecker from_secret vars) at provisioning time. provision-stack.py
# FAILS HARD if any placeholder is missing/empty.
# - The rendered traefik/traefik.env exists only on the host (gitignored).
# - Rendered by provision-stack.py, NOT Woodpecker's yaml preprocessor —
# single-dollar placeholders are safe here (deploy.yml's double-dollar
# rule does NOT apply to this file).
#
# Consumed by traefik/traefik.yaml (KEEPALIVED_PASSWORD, KEEPALIVED_VIRTUAL_IPS).
# DOMAIN comes from deploy/global.env, not here.
#
# ⚠️ HIGH RISK: Traefik is the entry point for all homelab services, and the
# Keepalived VIP (192.168.4.30) depends on this stack. KEEPALIVED_PASSWORD
# MUST be identical across keepalived-master and keepalived-backup — both
# consume this same rendered value — or VRRP auth fails and the VIP
# becomes unstable (this is exactly what the 2026-09-12 incident was:
# a committed literal "***REDACTED***" placeholder in git, restored every
# time the file was deleted/re-synced, diverging from -backup's stale but
# correct in-memory value).
#
# Classic VRRP simple-auth is silently truncated to 8 characters by
# keepalived itself. Set TRAEFIK_KEEPALIVED_PASSWORD in Woodpecker to a
# value 8 characters or fewer (or accept that only the first 8 chars are
# actually significant) so the effective negotiated value is unambiguous.
#
# 2026-09-12 RE-TRIGGER: traefik.env was found MISSING on disk after the
# Pattern C migration merged — the manifest-driven provisioning render only
# happens on a push that changes traefik/, and traefik is bootstrap-tier
# (stack-deploy.sh must be run manually; the pipeline never auto-deploys
# this stack). A manual `stack-deploy.sh traefik` had apparently been run
# while traefik.env was absent, causing Swarm's own interpolation to
# silently render KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS as EMPTY
# STRINGS on keepalived-master (not missing — empty, which is worse,
# since it's a valid-looking VRRP auth string that mismatches everything).
# This comment-only change re-triggers provision-secrets to re-render and
# re-ship traefik.env; a manual `stack-deploy.sh traefik` must still be run
# afterward to actually apply it to the running services.
# ─────────────────────────────────────────────────────────────────────────
# ── Keepalived (non-secret config) ────────────────────────────────────────
KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['192.168.4.30']"
# ── Keepalived (secret) ────────────────────────────────────────────────────
KEEPALIVED_PASSWORD=${TRAEFIK_KEEPALIVED_PASSWORD}