reverse-proxy runs Swarm mode:global (one instance per node), all
writing to the same CephFS file. This wrapper uses a shared flock +
shared logrotate state file (both also on the CephFS mount) so cron
on docker-1/2/3 can run independently without racing or double-rotating.
Traefik has no built-in log rotation and no SIGUSR1/reopen handling.
--accesslog.filePath and --log.filePath (traefik.yaml) write directly
to /volume1/docker/traefik/logs, which had grown to 15.5GB unrotated,
contributing to disk pressure (docker-2 CephFS at 82%) and adding
write latency to every request through Traefik.
traefik/traefik.env did not exist on the host after PR #21 merged. Root
cause: provisioning only renders on a push touching traefik/, and traefik
is bootstrap-tier (stack-deploy.sh runs manually, never via the pipeline).
A manual stack-deploy.sh traefik run apparently happened while the env file
was absent, causing Swarm's own interpolation to render
KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS as empty strings on
keepalived-master. This comment-only change re-triggers provision-secrets;
a manual stack-deploy.sh traefik run is still required afterward to apply
the freshly-rendered env to the running services.
Root cause of the 2026-09-12 VIP instability incident: traefik/traefik.env
was committed to git with a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD. Every git-guard resync/checkout restored that broken
value onto disk, diverging from keepalived-backup's stale-but-correct
in-memory value and causing a continuous VRRP auth failure + VIP flapping.
- Adds TRAEFIK_KEEPALIVED_PASSWORD (from_secret: traefik_keepalived_password)
to provision-secrets environment block
- Replaces the traefik legacy case-entry (previously grouped with
maintenance|media|unifi|... as "no Docker secrets, host .env only") with
a call to deploy/provision-stack.py traefik, matching the ai) stack's
manifest-driven pattern
- traefik/traefik.env is now rendered fresh from traefik/traefik.env.template
on every provisioning run and never committed to git again
Diffed $${ occurrence count against previous version before committing,
per this file's own 2026-08-26 incident note.
The committed version contained a literal "***REDACTED***" placeholder as
KEEPALIVED_PASSWORD, which was restored onto disk every time the real file
was deleted or the local checkout re-synced from git. This caused VRRP auth
to mismatch between keepalived-master and keepalived-backup (master picked
up the broken committed value, backup retained a stale-but-correct value),
producing a continuous "(VI_1) received an invalid passwd!" flood and VIP
instability. traefik.env is now rendered fresh from traefik.env.template by
deploy/provision-stack.py on every provisioning run and never committed.
Camera cards using advanced-camera-card were stuck retrying WebRTC ICE
negotiation every few seconds (visible as dashboards "refreshing").
frigate-nvr had no published ports at all, so go2rtc's WebRTC listener
on 8555 was unreachable from browsers despite valid ICE candidates
already configured (VIP + CIDR + stun) in frigate/config.yml.
Publishing via ingress mode keeps this working regardless of which
swarm node Frigate lands on (no placement constraint on this service).
No Traefik/auth changes - HA<->Frigate signaling was already working
correctly over traefik_backend; only the browser<->go2rtc media path
was broken.
Found via live testing on docker-2 after merging #19: git-guard.sh is
tracked in git at mode 100644 (not executable). stack-deploy.sh correctly
invokes it as `bash deploy/git-guard.sh`, sidestepping the exec bit for the
first call -- but the script's own internal `exec "$0" "$@"` calls (used to
cleanly re-run after a successful push or stash-pop) rely on the kernel
executing the file directly, which requires +x. Since every fresh checkout
or fast-forward preserves the tracked 644 mode, this failed with
"Permission denied" (exit 126) the moment either self-re-invocation path
was actually exercised.
Confirmed via a live dry run: simulated the exact #18 incident shape (dirty
git-guard.sh on a HEAD 3 commits behind origin/main) on docker-2. The new
stash-first logic from #19 worked perfectly end-to-end -- detected
dirty+stale, stashed safely, fast-forwarded via resync_with_origin(), and
popped the stash cleanly -- but then hit this pre-existing bug on the
final `exec "$0" "$@"` re-invocation. This bug pre-dates #19 (the old
dirty-commit-then-push-success path had the identical pattern); #19 just
added a second trigger point that happened to surface it during testing.
Fix: `exec bash "$0" "$@"` explicitly invokes through the interpreter
instead of relying on the file's own execute bit -- correct regardless of
what git tracks the file's mode as.
Adds the unmerged-conflict guard flagged in PR review: before the dirty-tree
flow can run, check `git ls-files -u` and refuse to auto-commit if unresolved
merge conflict markers are present (e.g. left behind by a prior run's failed
`git stash pop`). Without this, a re-run's `git add -A` would silently stage
literal <<<<<<< / ======= / >>>>>>> markers into a real commit and push them
to origin/main.
Also expands every failure-path message (stash-push failure, stash-pop
conflict vs. untracked-file-collision, resync failure, true divergence,
manual-abort cases) to include concrete, copy-pasteable remediation command
sequences with multiple options (resolve-in-place vs. abandon-and-reapply vs.
drop-if-unneeded), rather than a single generic hint. The true-divergence
message now also tells the operator to check `git stash list` first, so a
stash created by the new dirty+stale path isn't confused with a second
manually-created one.
No change to the common dirty-but-current or clean-but-stale behavior.
Previously the dirty-tree branch committed unconditionally, then only
discovered staleness/divergence when the push was rejected -- leaving a
doomed auto-commit sitting on a stale base. Root-caused from a real
incident: a stray on-disk edit to ai/ai.yaml sat on a checkout that was
already 4 commits behind, git-guard auto-committed on top of it, then
the push was rejected (ahead 1, behind 4), aborting the deploy.
Fix: when DIRTY=1 and LOCAL != REMOTE, stash the dirty changes first,
resync main with origin using the exact same behind/ahead/diverged
rules as the clean-tree path (now shared via resync_with_origin()),
then reapply the stash and re-run. Never commits on top of a stale
base again. On any failure the stash is preserved and printed for
manual recovery -- changes are never silently lost.
Adds the missing bind mount for FlowAgent's delegated-user MSAL disk
token cache:
/volume1/docker/mcpo/flowagent-auth:/app/flowagent-auth
Confirmed via `docker service inspect ai_mcpo --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'`-equivalent review of the current
compose that mcpo's mounts were only: ssh_keys, config.json, mcpo_data,
uv-cache -- flowagent-auth (set as HOME for the flowagent subprocess via
config.json.template's env.HOME=/app/flowagent-auth) was NOT persisted,
so any successful delegated-user MSAL sign-in (list_connections /
create_connection / resolve_refs / pick_or_create_connection /
resolve_params -- the tools that use MsalTokenProvider, see companion
homelab/mcp-config PR #9) would be silently lost on every mcpo restart,
forcing a fresh interactive sign-in every time.
Host dir already exists (pre-created on shared CephFS, identical across
all three docker LXCs):
mkdir -p /volume1/docker/mcpo/flowagent-auth
This does NOT change the flowagent image tag/pin -- that stays
git.bryanmail.net/homelab/flowagent-mcpo:c5b22618404a pending the
companion mcp-config PR merging and its Woodpecker build producing a new
commit-sha tag to bump to.
No other changes to ai.yaml in this commit -- diff is mount-list addition
only.
Root cause of PR #16's pipeline failure (#398-#401): secrets/ is a
tooling folder (secrets-map.yaml, *.secrets.example), not a stack, but
folder-detection (cut -d/ -f1, excluding only deploy/) computed
ALL_STACKS="ai secrets" the first time a commit touched secrets/. The
deploy step survived only because 'secrets' coincidentally sits in the
bootstrap-tier skip list; verify had no guard — `docker stack ps
secrets` exited non-zero and, under errexit, the OUTPUT=$(...)
assignment killed the step before the designed WARNING path could run.
The ai deploy itself was fully successful (complete 21-key render, all
6 services healthy) — the failure was pure phantom. Hazard first
flagged July 2026 in PR #3 (closed unmerged).
Changes:
- grep -v '^deploy$' -> grep -vE '^(deploy|secrets)$' in ALL five
folder-detection sites (validate, provision-secrets x2, deploy,
verify)
- verify: `|| true` inside the stack-ps command substitution so a
genuinely missing stack warns instead of aborting (comment added)
- deploy step: also rsync secrets/ tooling to the host mirror
unconditionally, same treatment as deploy/
- header: 2026-09-08 incident note
Everything else byte-identical to main. Dollar-escape audit done.
Same two scoped changes as PR #15's rebase, now applied on a branch cut
from TODAY's main so the merge base is current (PR #15 became
unmergeable — its 09-03 merge base predated main's heavy hand-edits):
1. ai) case -> `python3 deploy/provision-stack.py ai` (one line). Retires
main's heredoc with its live drift: AI_LITELLM_POSTGRES_PASSWORD
printed twice (one copy from retired AI_LITELLM_DB_PASSWORD), stray
legacy AI_WEBUI_SECRET_KEY, and the WEB_UI/WEBUI shell-var mismatch
that renders an EMPTY OAuth client secret into ai.env.
2. env block ai section: normalized to exact template placeholder names,
added AI_MCPO_API_KEY (NEW Woodpecker secret required pre-merge),
pruned now-template-literal vars.
apk gains python3 py3-yaml. Everything else byte-matches current main.
Dollar-escape audit done per header lesson.
Carried over from secrets-provisioning-v2 (PR #15), content identical.
Whole-file template render (hard-fail naming missing vars), env + Docker
secrets shipped via ssh stdin only, sha256-checksum skip-if-unchanged,
never prints a value. Stacks absent from the manifest exit 0 so legacy
case-entries keep handling them.
Carried over from secrets-provisioning-v2 (PR #15) post-rebase. Key names
match exactly what current ai/ai.yaml references; MCPO_API_KEY stays
unprefixed (ai.yaml consumes it unprefixed for both mcpo services).
Placeholder names match the CI env vars declared in deploy.yml's
provision-secrets block, including the normalized
AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET (fixing main's WEB_UI/WEBUI mismatch
that currently renders an EMPTY OAuth client secret into ai.env).
NEW Woodpecker secret required before merge: ai_mcpo_api_key.
Carried over from secrets-provisioning-v2 (PR #15), which became
unmergeable after heavy parallel hand-edits to main moved the merge base
too far. Content identical to v2.
Surgical addition to the existing case "ai)" block only — every other
case, comment, and line in this ~26KB pipeline file is byte-for-byte
unchanged from main. Given this file's documented incident history from
prior full-file rewrites (see header comment), this was hand-verified
against the current main content line-by-line before writing, not
generated fresh.
Adds:
- 3 new environment: entries (FLOWAGENT_AZURE_CLIENT_ID/TENANT_ID/
CLIENT_SECRET) via from_secret, alongside the existing ai_* secrets
- Inside the existing `ai)` case, after the ai.env rewrite block: three
create_or_update_secret calls provisioning flowagent_azure_client_id,
flowagent_azure_tenant_id, flowagent_azure_client_secret as native
Docker Swarm secrets (Pattern C, _FILE convention) — NOT written into
ai/ai.env, since mcpo's flowagent entry reads them via
/run/secrets/flowagent_azure_* per flowagent/entrypoint.sh, not env vars
- A short comment explaining why these three are Pattern C secrets
inside an otherwise Pattern-B (host .env) stack case
This is the matched-pair companion to the ai.yaml commit in this same PR,
per the FlowAgent progress note's explicit sequencing requirement (both
land together, reviewed, not auto-merged to main).
Image pinned to git.bryanmail.net/homelab/flowagent-mcpo:916164714429 —
the exact sha-tagged image built and smoke-tested clean in mcp-config
pipeline #22 (build-and-push-flowagent + smoke-test-flowagent-image both
success). NOT :latest, for reproducible deploys.
Adds a secrets: block to the mcpo service only, referencing the 3
flowagent_azure_* Docker Swarm secrets (Pattern C, _FILE convention,
consumed by flowagent/entrypoint.sh via /run/secrets/flowagent_azure_*).
Declares those 3 secrets as external at top level, mirroring the existing
vaultwarden_admin_token_v2 pattern in vaultwarden.yaml.
Companion secret-provisioning change lands in the same PR as a separate
commit in .woodpecker/deploy.yml's provision-secrets step (case "ai)").
mcpo-critical (lines defining Proxmox access) is completely untouched —
still stock ghcr.io/open-webui/mcpo:main, no secrets: block, unchanged.
Per FlowAgent MCP progress note: Azure AD App Registration is reported
created with real secrets already in Woodpecker (both mcp-config and
compose-files repos) as of this PR. This is a reviewed PR against a
feature branch, NOT auto-merged to main — deploy only happens after
manual review/merge, per user's explicit request for a manual-revert
safety net on this high-blast-radius shared stack (LiteLLM, Open WebUI,
n8n also live here).
Context: git hosts the source of truth for every other stack's compose
files, so in a disaster-recovery scenario it must be restorable from a
flat git.yaml + git.env backup alone, with zero dependency on a running
Swarm's Docker secret store (native Docker secrets can't be resolved
until Swarm already exists, which is the circular dependency git.yaml
would otherwise create). Path 2 (documented decision): Pattern B for
git specifically, same mechanism the ai stack already uses.
provision-secrets / git) case:
- Previously created native Docker secrets (git_db_password,
git_runner_token, git_mcp_access_token) via create-secrets.sh —
leftover from an earlier, abandoned Pattern C attempt.
- Now rewrites only the secret-bearing lines (GITEA__database__PASSWD,
GITEA_RUNNER_REGISTRATION_TOKEN, GITEA_MCP_ACCESS_TOKEN) via
grep -v + printf, mirroring the ai) case exactly. Also strips the
legacy GITEA_ACCESS_TOKEN key name so the test file converges on the
git.env.example-documented key.
- TEST PHASE: target is git.env.pipelinetest, NOT git.env. The real
git.env is never opened for writing by this step. First run seeds
the test file from the real git.env (to carry over all non-secret
lines), then only the 3 secret lines are refreshed on every push.
- Real cutover (pointing git.yaml/stack-deploy at the generated file,
then retiring git.env.pipelinetest) is a deliberate follow-up step
after manually diffing the rendered output.
deploy / bootstrap-tier guard:
- Added git to the traefik|woodpecker|postgresql|secrets guard list.
git.yaml changes now sync to the host but require a manual
`stack-deploy.sh git` run, same as the other foundational stacks —
prevents an auto-deploy of a bad git.yaml change from taking down
Gitea before a human can look at it (Gitea itself is what every
other pipeline needs to trigger a fix).
Bot