Flags may appear before or after the stack name (stack-deploy.sh traefik -e
and stack-deploy.sh -e traefik both work) — the stack name is whatever
argument isn't a recognized flag. No change to the render/mount-guard/deploy
logic; this only changes how the git-guard.sh pre-flight is invoked. See
deploy/git-guard.sh header for what each flag actually does.
Adds two opt-in flags to break the circular dependency discovered during
the 2026-09-12 traefik/keepalived incident: Traefik down -> VIP/hostname
unreachable -> git-guard can't fetch -> stack-deploy.sh traefik blocked ->
can't redeploy traefik to fix Traefik.
-e/--emergency: probes git.bryanmail.net, then the VIP (192.168.4.30), then
each node's direct IP (.31/.32/.33) on Gitea's direct ingress port 3000
(bypasses Traefik/VIP entirely), switches origin to the first reachable one,
then runs the NORMAL sync logic against it (behind/ahead/diverged handling
unchanged — this only changes which host is used, never skips the safety
checks). Prints a reminder to restore the real origin URL afterward; never
persists the swap anywhere.
-f/--force: skips the sync check entirely, for the genuine last-resort case
where Gitea itself (not just routing) is unreachable. Loud warning banner.
Verified git ls-remote succeeds unauthenticated over plain http against
each candidate before writing this, so no credential-smuggling concern.
reverse-proxy runs Swarm mode:global (one instance per node), all
writing to the same CephFS file. This wrapper uses a shared flock +
shared logrotate state file (both also on the CephFS mount) so cron
on docker-1/2/3 can run independently without racing or double-rotating.
Traefik has no built-in log rotation and no SIGUSR1/reopen handling.
--accesslog.filePath and --log.filePath (traefik.yaml) write directly
to /volume1/docker/traefik/logs, which had grown to 15.5GB unrotated,
contributing to disk pressure (docker-2 CephFS at 82%) and adding
write latency to every request through Traefik.
traefik/traefik.env did not exist on the host after PR #21 merged. Root
cause: provisioning only renders on a push touching traefik/, and traefik
is bootstrap-tier (stack-deploy.sh runs manually, never via the pipeline).
A manual stack-deploy.sh traefik run apparently happened while the env file
was absent, causing Swarm's own interpolation to render
KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS as empty strings on
keepalived-master. This comment-only change re-triggers provision-secrets;
a manual stack-deploy.sh traefik run is still required afterward to apply
the freshly-rendered env to the running services.
Root cause of the 2026-09-12 VIP instability incident: traefik/traefik.env
was committed to git with a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD. Every git-guard resync/checkout restored that broken
value onto disk, diverging from keepalived-backup's stale-but-correct
in-memory value and causing a continuous VRRP auth failure + VIP flapping.
- Adds TRAEFIK_KEEPALIVED_PASSWORD (from_secret: traefik_keepalived_password)
to provision-secrets environment block
- Replaces the traefik legacy case-entry (previously grouped with
maintenance|media|unifi|... as "no Docker secrets, host .env only") with
a call to deploy/provision-stack.py traefik, matching the ai) stack's
manifest-driven pattern
- traefik/traefik.env is now rendered fresh from traefik/traefik.env.template
on every provisioning run and never committed to git again
Diffed $${ occurrence count against previous version before committing,
per this file's own 2026-08-26 incident note.
The committed version contained a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD, which was restored onto disk every time the real file
was deleted or the local checkout re-synced from git. This caused VRRP auth
to mismatch between keepalived-master and keepalived-backup (master picked
up the broken committed value, backup retained a stale-but-correct value),
producing a continuous "(VI_1) received an invalid passwd!" flood and VIP
instability. traefik.env is now rendered fresh from traefik.env.template by
deploy/provision-stack.py on every provisioning run and never committed.
Camera cards using advanced-camera-card were stuck retrying WebRTC ICE
negotiation every few seconds (visible as dashboards "refreshing").
frigate-nvr had no published ports at all, so go2rtc's WebRTC listener
on 8555 was unreachable from browsers despite valid ICE candidates
already configured (VIP + CIDR + stun) in frigate/config.yml.
Publishing via ingress mode keeps this working regardless of which
swarm node Frigate lands on (no placement constraint on this service).
No Traefik/auth changes - HA<->Frigate signaling was already working
correctly over traefik_backend; only the browser<->go2rtc media path
was broken.
Found via live testing on docker-2 after merging #19: git-guard.sh is
tracked in git at mode 100644 (not executable). stack-deploy.sh correctly
invokes it as `bash deploy/git-guard.sh`, sidestepping the exec bit for the
first call -- but the script's own internal `exec "$0" "$@"` calls (used to
cleanly re-run after a successful push or stash-pop) rely on the kernel
executing the file directly, which requires +x. Since every fresh checkout
or fast-forward preserves the tracked 644 mode, this failed with
"Permission denied" (exit 126) the moment either self-re-invocation path
was actually exercised.
Confirmed via a live dry run: simulated the exact #18 incident shape (dirty
git-guard.sh on a HEAD 3 commits behind origin/main) on docker-2. The new
stash-first logic from #19 worked perfectly end-to-end -- detected
dirty+stale, stashed safely, fast-forwarded via resync_with_origin(), and
popped the stash cleanly -- but then hit this pre-existing bug on the
final `exec "$0" "$@"` re-invocation. This bug pre-dates #19 (the old
dirty-commit-then-push-success path had the identical pattern); #19 just
added a second trigger point that happened to surface it during testing.
Fix: `exec bash "$0" "$@"` explicitly invokes through the interpreter
instead of relying on the file's own execute bit -- correct regardless of
what git tracks the file's mode as.
Adds the unmerged-conflict guard flagged in PR review: before the dirty-tree
flow can run, check `git ls-files -u` and refuse to auto-commit if unresolved
merge conflict markers are present (e.g. left behind by a prior run's failed
`git stash pop`). Without this, a re-run's `git add -A` would silently stage
literal <<<<<<< / ======= / >>>>>>> markers into a real commit and push them
to origin/main.
Also expands every failure-path message (stash-push failure, stash-pop
conflict vs. untracked-file-collision, resync failure, true divergence,
manual-abort cases) to include concrete, copy-pasteable remediation command
sequences with multiple options (resolve-in-place vs. abandon-and-reapply vs.
drop-if-unneeded), rather than a single generic hint. The true-divergence
message now also tells the operator to check `git stash list` first, so a
stash created by the new dirty+stale path isn't confused with a second
manually-created one.
No change to the common dirty-but-current or clean-but-stale behavior.
Previously the dirty-tree branch committed unconditionally, then only
discovered staleness/divergence when the push was rejected -- leaving a
doomed auto-commit sitting on a stale base. Root-caused from a real
incident: a stray on-disk edit to ai/ai.yaml sat on a checkout that was
already 4 commits behind, git-guard auto-committed on top of it, then
the push was rejected (ahead 1, behind 4), aborting the deploy.
Fix: when DIRTY=1 and LOCAL != REMOTE, stash the dirty changes first,
resync main with origin using the exact same behind/ahead/diverged
rules as the clean-tree path (now shared via resync_with_origin()),
then reapply the stash and re-run. Never commits on top of a stale
base again. On any failure the stash is preserved and printed for
manual recovery -- changes are never silently lost.
Adds the missing bind mount for FlowAgent's delegated-user MSAL disk
token cache:
/volume1/docker/mcpo/flowagent-auth:/app/flowagent-auth
Confirmed via `docker service inspect ai_mcpo --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'`-equivalent review of the current
compose that mcpo's mounts were only: ssh_keys, config.json, mcpo_data,
uv-cache -- flowagent-auth (set as HOME for the flowagent subprocess via
config.json.template's env.HOME=/app/flowagent-auth) was NOT persisted,
so any successful delegated-user MSAL sign-in (list_connections /
create_connection / resolve_refs / pick_or_create_connection /
resolve_params -- the tools that use MsalTokenProvider, see companion
homelab/mcp-config PR #9) would be silently lost on every mcpo restart,
forcing a fresh interactive sign-in every time.
Host dir already exists (pre-created on shared CephFS, identical across
all three docker LXCs):
mkdir -p /volume1/docker/mcpo/flowagent-auth
This does NOT change the flowagent image tag/pin -- that stays
git.bryanmail.net/homelab/flowagent-mcpo:c5b22618404a pending the
companion mcp-config PR merging and its Woodpecker build producing a new
commit-sha tag to bump to.
No other changes to ai.yaml in this commit -- diff is mount-list addition
only.
Root cause of PR #16's pipeline failure (#398-#401): secrets/ is a
tooling folder (secrets-map.yaml, *.secrets.example), not a stack, but
folder-detection (cut -d/ -f1, excluding only deploy/) computed
ALL_STACKS="ai secrets" the first time a commit touched secrets/. The
deploy step survived only because 'secrets' coincidentally sits in the
bootstrap-tier skip list; verify had no guard — `docker stack ps
secrets` exited non-zero and, under errexit, the OUTPUT=$(...)
assignment killed the step before the designed WARNING path could run.
The ai deploy itself was fully successful (complete 21-key render, all
6 services healthy) — the failure was pure phantom. Hazard first
flagged July 2026 in PR #3 (closed unmerged).
Changes:
- grep -v '^deploy$' -> grep -vE '^(deploy|secrets)$' in ALL five
folder-detection sites (validate, provision-secrets x2, deploy,
verify)
- verify: `|| true` inside the stack-ps command substitution so a
genuinely missing stack warns instead of aborting (comment added)
- deploy step: also rsync secrets/ tooling to the host mirror
unconditionally, same treatment as deploy/
- header: 2026-09-08 incident note
Everything else byte-identical to main. Dollar-escape audit done.
Same two scoped changes as PR #15's rebase, now applied on a branch cut
from TODAY's main so the merge base is current (PR #15 became
unmergeable — its 09-03 merge base predated main's heavy hand-edits):
1. ai) case -> `python3 deploy/provision-stack.py ai` (one line). Retires
main's heredoc with its live drift: AI_LITELLM_POSTGRES_PASSWORD
printed twice (one copy from retired AI_LITELLM_DB_PASSWORD), stray
legacy AI_WEBUI_SECRET_KEY, and the WEB_UI/WEBUI shell-var mismatch
that renders an EMPTY OAuth client secret into ai.env.
2. env block ai section: normalized to exact template placeholder names,
added AI_MCPO_API_KEY (NEW Woodpecker secret required pre-merge),
pruned now-template-literal vars.
apk gains python3 py3-yaml. Everything else byte-matches current main.
Dollar-escape audit done per header lesson.
Carried over from secrets-provisioning-v2 (PR #15), content identical.
Whole-file template render (hard-fail naming missing vars), env + Docker
secrets shipped via ssh stdin only, sha256-checksum skip-if-unchanged,
never prints a value. Stacks absent from the manifest exit 0 so legacy
case-entries keep handling them.
Carried over from secrets-provisioning-v2 (PR #15) post-rebase. Key names
match exactly what current ai/ai.yaml references; MCPO_API_KEY stays
unprefixed (ai.yaml consumes it unprefixed for both mcpo services).
Placeholder names match the CI env vars declared in deploy.yml's
provision-secrets block, including the normalized
AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET (fixing main's WEB_UI/WEBUI mismatch
that currently renders an EMPTY OAuth client secret into ai.env).
NEW Woodpecker secret required before merge: ai_mcpo_api_key.
Carried over from secrets-provisioning-v2 (PR #15), which became
unmergeable after heavy parallel hand-edits to main moved the merge base
too far. Content identical to v2.
AVB
Bot