fix(git-guard): use 'exec bash "$0"' instead of 'exec "$0"' for self-re-invocation

Found via live testing on docker-2 after merging #19: git-guard.sh is
tracked in git at mode 100644 (not executable). stack-deploy.sh correctly
invokes it as `bash deploy/git-guard.sh`, sidestepping the exec bit for the
first call -- but the script's own internal `exec "$0" "$@"` calls (used to
cleanly re-run after a successful push or stash-pop) rely on the kernel
executing the file directly, which requires +x. Since every fresh checkout
or fast-forward preserves the tracked 644 mode, this failed with
"Permission denied" (exit 126) the moment either self-re-invocation path
was actually exercised.

Confirmed via a live dry run: simulated the exact #18 incident shape (dirty
git-guard.sh on a HEAD 3 commits behind origin/main) on docker-2. The new
stash-first logic from #19 worked perfectly end-to-end -- detected
dirty+stale, stashed safely, fast-forwarded via resync_with_origin(), and
popped the stash cleanly -- but then hit this pre-existing bug on the
final `exec "$0" "$@"` re-invocation. This bug pre-dates #19 (the old
dirty-commit-then-push-success path had the identical pattern); #19 just
added a second trigger point that happened to surface it during testing.

Fix: `exec bash "$0" "$@"` explicitly invokes through the interpreter
instead of relying on the file's own execute bit -- correct regardless of
what git tracks the file's mode as.

.woodpecker/deploy.yml

@@ -48,6 +48,28 @@ when:
 #   already unconditionally rsynced at the top of the deploy step regardless
 #   of which stacks changed, so it's safe to exclude it from the stack list
 #   everywhere folders are detected below.
+#
+#   2026-09-03/07 INCIDENT + REDESIGN: hand-maintained grep -v + printf
+#   line-surgery cases repeatedly drifted (missing '=', duplicated keys,
+#   mismatched env var names rendering EMPTY secrets) and broke live
+#   services. Root cause: no authoritative key list and shell heredocs
+#   hostile to hand-editing. Stacks migrate one at a time to a data-driven
+#   model: secrets/secrets-map.yaml (data only) + per-stack .env.template
+#   (authoritative FULL file) + deploy/provision-stack.py (whole-file
+#   render, hard failure naming any missing value). Migrated stacks call
+#   the script; unmigrated stacks keep legacy case entries until their own
+#   PR. See the "Secrets & Deployment Architecture — Global Direction" note.
+#
+#   2026-09-08 FIX: secrets/ is a tooling/docs folder (secrets-map.yaml +
+#   *.secrets.example), not a stack — but folder-detection treated it as one
+#   the first time a commit touched it (PR #16). deploy survived only because
+#   'secrets' sits in the bootstrap-tier skip list; verify had no guard and
+#   died on `docker stack ps secrets` failing under errexit (assignment from
+#   a failing command substitution aborts the step). Fixed by excluding
+#   secrets/ alongside deploy/ in ALL folder-detection sites, and by
+#   tolerating a failing stack-ps in verify (|| true) so a genuinely missing
+#   stack produces the designed WARNING instead of killing the step. This
+#   hazard was first flagged in July (PR #3, closed unmerged).
 # ─────────────────────────────────────────────────────────────────────────────
 
 steps:
@@ -65,9 +87,9 @@ steps:
         FLAT=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.ya?ml$' || true)
 
         # Folder: any file under a subfolder (e.g. immich/immich.yml).
-        # Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and deploy/
-        # (shared tooling, not a stack — see note above).
-        FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        # Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and the
+        # non-stack tooling folders deploy/ and secrets/ (see notes above).
+        FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
 
         [ -z "$FLAT" ] && [ -z "$FOLDERS" ] && echo "No stacks changed" && exit 0
 
@@ -163,6 +185,11 @@ steps:
         from_secret: entertainment_sparky_encryption_key
       ENTERTAINMENT_BETTER_AUTH_SECRET:
         from_secret: entertainment_better_auth_secret
+      # ── ai stack (manifest-driven — secrets/secrets-map.yaml +
+      #    ai/ai.env.template + deploy/provision-stack.py). Env var names
+      #    below match the template placeholders EXACTLY; this block is the
+      #    only per-secret touchpoint left in this file for migrated stacks
+      #    (Woodpecker v3 requires explicit from_secret declarations). ──
       AI_AWS_ACCESS_KEY_ID:
         from_secret: ai_aws_access_key_id
       AI_AWS_SECRET_ACCESS_KEY:
@@ -171,38 +198,18 @@ steps:
         from_secret: ai_litellm_master_key
       AI_LITELLM_SALT_KEY:
         from_secret: ai_litellm_salt_key
-      AI_LITELLM_DB_PASSWORD:
-        from_secret: ai_litellm_db_password
-      AI_LITELLM_DATABASE_MIGRATIONS:
-        from_secret: litellm_database_migrations
-      AI_LITELLM_MODIFY_PARAMS:
-        from_secret: litellm_modify_params
-      AI_OPEN_WEBUI_SECRET_KEY:
-        from_secret: ai_webui_secret_key
-      AI_OPEN_WEBUI_DATABASE_URL:
-        from_secret: ai_open_webui_database_url
-      AI_OPEN_WEB_UI_OAUTH_CLIENT_SECRET:
-        from_secret: ai_oauth_client_secret
-      AI_OPEN_WEB_UI_ENABLE_OAUTH_SIGNUP:
-        from_secret: ai_open_web_ui_enable_oauth_signup
-      AI_OPEN_WEB_UI_OAUTH_MERGE_ACCOUNTS_BY_EMAIL:
-        from_secret: ai_open_web_ui_oauth_merge_accounts_by_email
-      AI_OPEN_WEB_UI_OAUTH_PROVIDER_NAME:
-        from_secret: ai_open_web_ui_oauth_provider_name
-      AI_OPEN_WEB_UI_OPENID_PROVIDER_URL:
-        from_secret: ai_open_web_ui_openid_provider_url
-      AI_OPEN_WEB_UI_OAUTH_CLIENT_ID:
-        from_secret: ai_open_web_ui_oauth_client_id
-      AI_OPEN_WEB_UI_OAUTH_SCOPES:
-        from_secret: ai_open_web_ui_oauth_scopes
-      AI_OPEN_WEB_UI_OPENID_REDIRECT_URI:
-        from_secret: ai_open_web_ui_openid_redirect_uri
       AI_LITELLM_DATABASE_URL:
         from_secret: ai_litellm_database_url
       AI_LITELLM_POSTGRES_PASSWORD:
         from_secret: ai_litellm_postgres_password
-      AI_AWS_REGION_NAME:
-        from_secret: ai_aws_region_name
+      AI_OPEN_WEBUI_SECRET_KEY:
+        from_secret: ai_webui_secret_key
+      AI_OPEN_WEBUI_DATABASE_URL:
+        from_secret: ai_open_webui_database_url
+      AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET:
+        from_secret: ai_oauth_client_secret
+      AI_MCPO_API_KEY:
+        from_secret: ai_mcpo_api_key
       FLOWAGENT_AZURE_CLIENT_ID:
         from_secret: flowagent_azure_client_id
       FLOWAGENT_AZURE_TENANT_ID:
@@ -210,7 +217,7 @@ steps:
       FLOWAGENT_AZURE_CLIENT_SECRET:
         from_secret: flowagent_azure_client_secret
     commands:
-      - apk add --no-cache openssh-client
+      - apk add --no-cache openssh-client python3 py3-yaml
       - mkdir -p ~/.ssh
       - echo "$SSH_KEY" | base64 -d > ~/.ssh/id_rsa
       - chmod 600 ~/.ssh/id_rsa
@@ -223,14 +230,14 @@ steps:
       - |
         CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
         FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
         ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
         [ -z "$ALL_STACKS" ] && echo "No stacks changed, skipping" && exit 0
       - scp -o StrictHostKeyChecking=no deploy/create-secrets.sh root@$${SWARM_MANAGER_IP}:/tmp/cs.sh
       - |
         CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
         FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
         ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
 
         for STACK in $ALL_STACKS; do
@@ -257,9 +264,9 @@ steps:
               # dependency on a running Swarm's Docker secret store. Native Docker
               # secrets (Pattern C) can't satisfy that: they only exist inside an
               # already-running Swarm, which is exactly the circular dependency
-              # this stack can't have. Mirrors the ai) case's grep -v + printf
-              # rewrite-in-place approach, never sed (values may contain slash,
-              # dollar sign, ampersand).
+              # this stack can't have. Mirrors the retired ai) case's grep -v +
+              # printf rewrite-in-place approach, never sed (values may contain
+              # slash, dollar sign, ampersand).
               #
               # TEST PHASE: target is git.env.pipelinetest, NOT the real git.env.
               # The real file is never opened for writing by this step. First run
@@ -270,6 +277,8 @@ steps:
               # documented GITEA_MCP_ACCESS_TOKEN key. Cutover to the real file —
               # and pointing git.yaml/stack-deploy at it — is a deliberate,
               # separate follow-up after manually diffing this render.
+              # (Candidate for the secrets-map.yaml/provision-stack.py migration
+              # in its own PR; kept legacy for now.)
               ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/git.env.pipelinetest
                 TMP=\$FILE.tmp.\$\$
                 [ -f \$FILE ] || cp /volume1/docker/compose-files/git.env \$FILE
@@ -317,56 +326,15 @@ steps:
                 create_or_update_secret 'vaultwarden_admin_token'     '$${VAULTWARDEN_ADMIN_TOKEN}'
                 create_or_update_secret 'vaultwarden_database_url_v2' '$${VAULTWARDEN_DATABASE_URL}'";;
             ai)
-              # NOTE: ai stack uses Pattern B (host .env, not native Docker secrets) —
-              # LiteLLM/boto3 and Open WebUI don't support the _FILE convention for these
-              # vars. Instead of Docker secrets, we regenerate ONLY the migrated lines in
-              # the remote ai/ai.env in place via grep -v + printf (never sed, since values
-              # may contain slash, dollar sign, ampersand). All other lines — including
-              # MCPO_API_KEY, OAUTH_CLIENT_ID, WEBUI_URL, and other non-secret config — are
-              # left completely untouched. MCPO_API_KEY migration is deferred to a
-              # follow-up; this step never reads or writes it.
-              #
-              # IMPORTANT: ai.yaml's litellm service references ${AI_AWS_ACCESS_KEY_ID} /
-              # ${AI_AWS_SECRET_ACCESS_KEY} (AI_-prefixed) and renders them into the
-              # container as plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (see commit
-              # 37ed671a, "Change AWS keys to use Woodpecker Secrets"). So ai.env must be
-              # written with the AI_-prefixed key names, NOT the plain ones — writing
-              # plain AWS_ACCESS_KEY_ID here would leave ${AI_AWS_ACCESS_KEY_ID}
-              # unresolved at compose-render time (empty), silently breaking Bedrock auth.
-              # All other migrated vars in ai.yaml use plain (unprefixed) names, so only
-              # these two lines need the AI_ prefix.
-              #
-              # FLOWAGENT NOTE (added alongside the ai.yaml mcpo image/secrets cutover):
-              # the 3 flowagent_azure_* values are provisioned as native Docker secrets
-              # below (Pattern C, matches every other _FILE-convention stack), NOT written
-              # into ai/ai.env — mcpo's flowagent entry reads them via
-              # /run/secrets/flowagent_azure_* (see mcp-config/flowagent/entrypoint.sh),
-              # not via env var, so they don't belong in this stack's Pattern B .env block.
-              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/ai/ai.env
-                TMP=\$FILE.tmp.\$\$
-                grep -vE '^(AI_AWS_ACCESS_KEY_ID|AI_AWS_SECRET_ACCESS_KEY|AI_LITELLM_MASTER_KEY|AI_LITELLM_SALT_KEY|AI_LITELLM_POSTGRES_PASSWORD|AI_LITELLM_DATABASE_URL|AI_OPEN_WEBUI_SECRET_KEY|AI_OPEN_WEBUI_DATABASE_URL|AI_OPEN_WEB_UI_OAUTH_CLIENT_SECRET)=' \$FILE > \$TMP 2>/dev/null || touch \$TMP
-                { cat \$TMP
-                  printf 'AI_AWS_ACCESS_KEY_ID=%s\n' '$${AI_AWS_ACCESS_KEY_ID}'
-                  printf 'AI_AWS_SECRET_ACCESS_KEY=%s\n' '$${AI_AWS_SECRET_ACCESS_KEY}'
-                  printf 'AI_AWS_REGION_NAME=%s\n' '$${AI_AWS_REGION_NAME}'
-                  printf 'AI_LITELLM_MASTER_KEY=%s\n' '$${AI_LITELLM_MASTER_KEY}'
-                  printf 'AI_LITELLM_SALT_KEY=%s\n' '$${AI_LITELLM_SALT_KEY}'
-                  printf 'AI_LITELLM_POSTGRES_PASSWORD=%s\n' '$${AI_LITELLM_DB_PASSWORD}'
-                  printf 'AI_LITELLM_DATABASE_URL=%s\n' '$${AI_LITELLM_DATABASE_URL}'
-                  printf 'AI_LITELLM_POSTGRES_PASSWORD=%s\n' '$${AI_LITELLM_POSTGRES_PASSWORD}'
-                  printf 'AI_LITELLM_MODIFY_PARAMS=%s\n' '$${AI_LITELLM_MODIFY_PARAMS}'
-                  printf 'AI_LITELLM_DATABASE_MIGRATIONS=%s\n' '$${AI_LITELLM_DATABASE_MIGRATIONS}'
-                  printf 'AI_OPEN_WEBUI_SECRET_KEY=%s\n' '$${AI_OPEN_WEBUI_SECRET_KEY}'
-                  printf 'AI_OPEN_WEBUI_DATABASE_URL=%s\n' '$${AI_OPEN_WEBUI_DATABASE_URL}'
-                  printf 'AI_WEBUI_SECRET_KEY=%s\n' '$${AI_OPEN_WEBUI_SECRET_KEY}'
-                  printf 'AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET=%s\n' '$${AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET}'
-                } > \$FILE
-                rm -f \$TMP
-                echo '  [OK] ai/ai.env secrets updated'
-                source /tmp/cs.sh
-                create_or_update_secret 'flowagent_azure_client_id'     '$${FLOWAGENT_AZURE_CLIENT_ID}'
-                create_or_update_secret 'flowagent_azure_tenant_id'     '$${FLOWAGENT_AZURE_TENANT_ID}'
-                create_or_update_secret 'flowagent_azure_client_secret' '$${FLOWAGENT_AZURE_CLIENT_SECRET}'";;
+              # MIGRATED (2026-09-07) to manifest-driven provisioning after the
+              # line-surgery approach repeatedly drifted (dropped keys 2026-09-03;
+              # duplicate AI_LITELLM_POSTGRES_PASSWORD + stray legacy keys +
+              # WEB_UI/WEBUI env-name mismatch rendering an EMPTY OAuth client
+              # secret, found 2026-09-07). All logic lives in
+              # deploy/provision-stack.py; the authoritative key list lives in
+              # ai/ai.env.template; the mapping lives in secrets/secrets-map.yaml.
+              # This case is intentionally one line.
+              python3 deploy/provision-stack.py ai;;
             entertainment)
               ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
                 create_or_update_secret 'entertainment_discord_token'          '$${ENTERTAINMENT_DISCORD_TOKEN}'
@@ -403,7 +371,7 @@ steps:
       - |
         CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
         FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
         ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
         [ -z "$ALL_STACKS" ] && echo "No stacks changed" && exit 0
 
@@ -411,6 +379,13 @@ steps:
         rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
           deploy/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/deploy/
 
+        # Sync secrets/ tooling (manifest + examples) alongside deploy/ —
+        # provision-stack.py reads secrets/secrets-map.yaml from the CI
+        # checkout, but the host mirror should stay complete for emergency
+        # manual provisioning runs.
+        rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
+          secrets/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/secrets/
+
         for STACK in $ALL_STACKS; do
           echo "--- Deploying: $STACK ---"
           # Sync files to host first (always, even for bootstrap stacks)
@@ -457,7 +432,7 @@ steps:
       - |
         CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
         FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
         ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
         [ -z "$ALL_STACKS" ] && exit 0
 
@@ -468,6 +443,10 @@ steps:
         # "nothing found in stack" output on ordinary deploys (e.g. vaultwarden,
         # 2026-08-25). Retry with backoff instead of a single fixed sleep, and
         # only warn (don't fail the pipeline) if tasks never show up.
+        # 2026-09-08: `|| true` inside the command substitution is REQUIRED —
+        # this step runs under errexit, and an assignment from a failing
+        # command substitution (e.g. `docker stack ps` on a stack that doesn't
+        # exist) kills the whole step before the WARNING path can run.
         ATTEMPTS=6
         DELAY=5
         for STACK in $ALL_STACKS; do
@@ -476,7 +455,7 @@ steps:
           while [ "$i" -le "$ATTEMPTS" ]; do
             OUTPUT=$(ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
               "docker stack ps $STACK --filter desired-state=running \
-              --format '  {{.Name}} {{.CurrentState}}'" 2>/dev/null)
+              --format '  {{.Name}} {{.CurrentState}}'" 2>/dev/null || true)
             if [ -n "$OUTPUT" ]; then
               echo "$OUTPUT"
               break

ai/ai.env.template

@@ -0,0 +1,55 @@
+# ─────────────────────────────────────────────────────────────────────────────
+# ai.env.template — AUTHORITATIVE template for ai/ai.env (rendered by
+# deploy/provision-stack.py per secrets/secrets-map.yaml).
+#
+#   - This file IS the complete key list for ai.env. The whole file is
+#     rendered on every provisioning run — no line surgery, so a key can
+#     never silently go missing again (root cause of the 2026-09-03 outage).
+#   - Key names match EXACTLY what ai/ai.yaml references (AI_<SERVICE>_*
+#     naming adopted on main 2026-09-06).
+#   - Non-secret config lives here as LITERAL values (visible, reviewable).
+#   - Secret values are dollar-brace placeholders resolved from the CI env
+#     (Woodpecker from_secret vars) at provisioning time. provision-stack.py
+#     FAILS HARD if any placeholder is missing/empty.
+#   - The rendered ai/ai.env exists only on the host (gitignored).
+#   - Rendered by provision-stack.py, NOT Woodpecker's yaml preprocessor —
+#     single-dollar placeholders are safe here (deploy.yml's double-dollar
+#     rule does NOT apply to this file).
+#
+# Consumed by ai/ai.yaml. DOMAIN_NAME comes from deploy/global.env, not here.
+# ─────────────────────────────────────────────────────────────────────────────
+
+# ── LiteLLM (non-secret config) ──────────────────────────────────────────────
+AI_AWS_REGION_NAME=us-east-2
+AI_LITELLM_MODIFY_PARAMS=False
+AI_LITELLM_DATABASE_MIGRATIONS=True
+
+# ── LiteLLM (secrets) ────────────────────────────────────────────────────────
+AI_AWS_ACCESS_KEY_ID=${AI_AWS_ACCESS_KEY_ID}
+AI_AWS_SECRET_ACCESS_KEY=${AI_AWS_SECRET_ACCESS_KEY}
+AI_LITELLM_MASTER_KEY=${AI_LITELLM_MASTER_KEY}
+AI_LITELLM_SALT_KEY=${AI_LITELLM_SALT_KEY}
+# Full connection URL is itself a secret (ai_litellm_database_url) — the
+# URL structure never appears in git.
+AI_LITELLM_DATABASE_URL=${AI_LITELLM_DATABASE_URL}
+AI_LITELLM_POSTGRES_PASSWORD=${AI_LITELLM_POSTGRES_PASSWORD}
+
+# ── Open WebUI (non-secret config) ───────────────────────────────────────────
+AI_OPEN_WEBUI_URL=https://ai.bryanmail.net
+AI_OPEN_WEBUI_ENABLE_OAUTH_SIGNUP=true
+AI_OPEN_WEBUI_OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
+AI_OPEN_WEBUI_OAUTH_PROVIDER_NAME=Authentik
+AI_OPEN_WEBUI_OPENID_PROVIDER_URL=https://auth.bryanmail.net/application/o/open-web-ui/.well-known/openid-configuration
+# OAuth client ID is a public identifier by OAuth2 design (it is sent to the
+# browser); the client SECRET below is the protected credential.
+AI_OPEN_WEBUI_OAUTH_CLIENT_ID=hVmhi1dS3TnG2cUw5QwLOx5FDLSWtnQUdZyeB5zK
+AI_OPEN_WEBUI_OAUTH_SCOPES=openid email profile
+AI_OPEN_WEBUI_OPENID_REDIRECT_URI=https://ai.bryanmail.net/oauth/oidc/callback
+
+# ── Open WebUI (secrets) ─────────────────────────────────────────────────────
+AI_OPEN_WEBUI_SECRET_KEY=${AI_OPEN_WEBUI_SECRET_KEY}
+AI_OPEN_WEBUI_DATABASE_URL=${AI_OPEN_WEBUI_DATABASE_URL}
+AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET=${AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET}
+
+# ── mcpo / mcpo-critical (secrets) ───────────────────────────────────────────
+MCPO_API_KEY=${AI_MCPO_API_KEY}

ai/ai.yaml

@@ -141,6 +141,7 @@ services:
     - /volume1/docker/mcpo/data:/mcpo_data
     - /volume1/docker/cronicle/ssh_keys:/app/ssh_keys:ro
     - /volume1/docker/mcpo/uv-cache:/app/uv-cache
+    - /volume1/docker/mcpo/flowagent-auth:/app/flowagent-auth
     secrets:
     - source: flowagent_azure_client_id
       target: flowagent_azure_client_id

deploy/git-guard.sh

@@ -5,10 +5,20 @@
 # Behavior:
 #   - Clean + up to date        -> pass silently
 #   - Clean + behind (ff-only)  -> auto `git pull --ff-only`, then pass
-#   - Ahead only (unpushed)     -> interactive: offer to push; non-interactive: BLOCK
-#   - Dirty tracked changes     -> offer to commit + push right now
-#                                   (auto in non-interactive/CI runs, after a
+#   - Clean + ahead only        -> interactive: offer to push; non-interactive: BLOCK
+#     (unpushed)
+#   - Unresolved merge conflict -> REFUSE immediately. Never auto-commits over
+#     markers present               conflict markers. Prints remediation options.
+#   - Dirty + local in sync     -> offer to commit + push right now
+#     with origin                 (auto in non-interactive/CI runs, after a
 #                                   secret-pattern scan of the staged diff)
+#   - Dirty + local STALE/      -> NEVER commit on top of a stale base. Stash
+#     diverged vs origin           the dirty changes first, resync main with
+#                                   origin using the same behind/ahead/diverged
+#                                   rules as the clean-tree case, then reapply
+#                                   the stash and re-run. On any failure the
+#                                   stash is preserved and remediation options
+#                                   (with exact commands) are printed.
 #   - Diverged (local AND       -> REFUSE. Never auto-resolves. Prints the
 #     remote both moved)           backup/stash/reset recovery steps and exits.
 #
@@ -36,12 +46,246 @@ git status --porcelain | grep -q . && DIRTY=1
 
 SECRET_PATTERN='(-----BEGIN [A-Z]+ PRIVATE KEY-----|AKIA[0-9A-Z]{16}|xox[baprs]-[0-9a-zA-Z-]+|password[[:space:]]*[:=][[:space:]]*[^$ ]|api[_-]?key[[:space:]]*[:=][[:space:]]*[^$ ])'
 
+# resync_with_origin <local_sha> <remote_sha> <base_sha>
+#
+# Handles the behind/ahead/diverged cases against a CLEAN working tree.
+# Shared by both the "tree was already clean" path and the new
+# "dirty tree turned out to be stale, so we stashed first" path, so the
+# two paths can never drift out of sync with each other.
+#
+# Returns 0 if it's now safe to deploy, 1 if it could not safely resolve
+# (guidance already printed to stdout in that case).
+resync_with_origin() {
+  local local_sha="$1" remote_sha="$2" base_sha="$3"
+
+  # ---- Case: fully in sync ----
+  if [ "$local_sha" = "$remote_sha" ]; then
+    echo "==> In sync with origin/main ($local_sha). OK to deploy."
+    return 0
+  fi
+
+  # ---- Case: behind only (fast-forwardable) ----
+  if [ "$local_sha" = "$base_sha" ]; then
+    echo "!! Local main is behind origin/main."
+    if [ "$INTERACTIVE" -eq 1 ]; then
+      read -rp "Fast-forward pull now? [y/N] " ans
+    else
+      ans="y"
+      echo "(non-interactive session — auto fast-forwarding)"
+    fi
+    if [[ "$ans" =~ ^[Yy]$ ]]; then
+      git pull --ff-only origin main
+      echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
+      return 0
+    else
+      echo "Aborting - pull manually, then retry:"
+      echo "  cd $DIR && git pull --ff-only origin main"
+      return 1
+    fi
+  fi
+
+  # ---- Case: ahead only (local commits not yet pushed) ----
+  if [ "$remote_sha" = "$base_sha" ]; then
+    echo "!! Local main is AHEAD of origin/main (unpushed commits):"
+    git log --oneline "origin/main..main"
+    echo
+    if [ "$INTERACTIVE" -eq 1 ]; then
+      read -rp "Push local commits to origin/main now? [y/N] " ans
+    else
+      ans="n"
+      echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
+    fi
+    if [[ "$ans" =~ ^[Yy]$ ]]; then
+      git push origin main
+      echo "==> Pushed. OK to deploy."
+      return 0
+    else
+      echo "Aborting. Review with:"
+      echo "  cd $DIR && git log origin/main..main"
+      echo "Then push manually when ready:"
+      echo "  git push origin main"
+      return 1
+    fi
+  fi
+
+  # ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
+  echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
+  echo "!! DIVERGED: local main and origin/main have both moved independently."
+  echo "!!"
+  echo "!!   Local-only commits:"
+  git log --oneline "$base_sha..main" | sed 's/^/!!     /'
+  echo "!!"
+  echo "!!   Remote-only commits:"
+  git log --oneline "$base_sha..origin/main" | sed 's/^/!!     /'
+  echo "!!"
+  echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
+  echo "!!"
+  echo "!! Before stashing anything new, check whether a git-guard safety stash"
+  echo "!! ALREADY exists from this same run (avoids confusing duplicate stashes):"
+  echo "!!   git stash list"
+  echo "!!"
+  echo "!! Recommended recovery:"
+  echo "!!   1. tar backup:      tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
+  echo "!!   2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
+  echo "!!   3. stash any NEW uncommitted state only if 'git stash list' above"
+  echo "!!      didn't already show one for this run:"
+  echo "!!        git stash push -u -m 'pre-reset-snapshot'"
+  echo "!!   4. reset to origin: git reset --hard origin/main"
+  echo "!!   5. selectively restore needed files from the stash/backup branch:"
+  echo "!!        git stash list"
+  echo "!!        git stash show -p stash@{N}"
+  echo "!!        git stash apply stash@{N}   # 'apply' keeps the stash as a backup; use 'pop' to also drop it"
+  echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
+  return 1
+}
+
+# ---- Case: unresolved merge conflict already present ----
+# Can happen if a PRIOR git-guard run's `git stash pop` conflicted and the
+# resulting conflict markers were never resolved before the next deploy
+# attempt. Must be checked BEFORE the dirty-tree commit flow below, because
+# an unmerged path shows up as "dirty" too, and `git add -A` would silently
+# stage the literal <<<<<<< / ======= / >>>>>>> markers into a real commit.
+if git ls-files -u | grep -q .; then
+  echo "ERROR: unresolved merge conflict markers present in the working tree."
+  echo "Refusing to auto-commit over a conflict — this would push literal"
+  echo "<<<<<<< / ======= / >>>>>>> markers to origin/main."
+  echo
+  echo "Conflicted paths:"
+  git diff --name-only --diff-filter=U | sed 's/^/  /'
+  echo
+  echo "Remediation options:"
+  echo "  A) Resolve the conflict by hand, then commit and push:"
+  echo "       cd $DIR"
+  echo "       git status                       # see conflicted paths"
+  echo "       git diff                          # inspect the conflict markers"
+  echo "       \$EDITOR <conflicted-file>          # remove markers, keep correct content"
+  echo "       git add <conflicted-file>"
+  echo "       git commit -m 'resolve git-guard stash-pop conflict'"
+  echo "       git push origin main"
+  echo "  B) Discard the conflicted merge attempt entirely and start clean from"
+  echo "     origin/main, then decide separately whether to re-apply anything"
+  echo "     from a prior safety stash:"
+  echo "       cd $DIR"
+  echo "       git checkout -- ."
+  echo "       git reset --hard origin/main"
+  echo "       git stash list                    # look for a git-guard-safety-stash-* entry"
+  echo "       git stash show -p stash@{N}         # inspect before deciding"
+  echo "  C) Once resolved (via A or B) and no longer needed, clean up the stash:"
+  echo "       git stash drop stash@{N}"
+  exit 1
+fi
+
 # ---- Case: dirty tracked changes ----
 if [ "$DIRTY" -eq 1 ]; then
   echo "!! WORKING TREE DIRTY — uncommitted changes detected:"
   git status --short
   echo
 
+  # If local is ALSO stale/diverged from origin, committing right now would
+  # create a doomed commit on top of a base that's about to be rejected on
+  # push (this is exactly what caused a real incident: a stray on-disk edit
+  # sat on a checkout that was 4 commits behind, git-guard auto-committed
+  # anyway, then the push bounced). Stash first, resync safely using the
+  # same rules as the clean-tree path, then reapply.
+  if [ "$LOCAL" != "$REMOTE" ]; then
+    echo "!! Local main is ALSO stale/diverged from origin/main."
+    echo "   Refusing to commit on top of a stale base — stashing the dirty"
+    echo "   changes safely first, then resyncing with origin."
+    echo
+
+    STASH_MSG="git-guard-safety-stash-$(date -u +%Y%m%dT%H%M%SZ)"
+    if ! git stash push -u -m "$STASH_MSG"; then
+      echo "ERROR: 'git stash push' itself failed (disk full, permissions, or"
+      echo "       some other git error). Your changes are still on disk,"
+      echo "       uncommitted — nothing has been lost, but git-guard cannot"
+      echo "       proceed safely until this is resolved."
+      echo
+      echo "Remediation options:"
+      echo "  A) Check disk space and permissions, then retry the deploy:"
+      echo "       df -h $DIR"
+      echo "       ls -la $DIR"
+      echo "  B) Identify and manually move aside whatever is blocking the stash,"
+      echo "     then retry:"
+      echo "       cd $DIR"
+      echo "       git status --short              # find the offending path(s)"
+      echo "       mv <path> <path>.bak-\$(date +%s)"
+      echo "  C) Inspect the raw git error above for specifics before proceeding."
+      exit 1
+    fi
+    echo "==> Stashed as: $STASH_MSG"
+
+    if resync_with_origin "$LOCAL" "$REMOTE" "$BASE"; then
+      echo "==> Resync succeeded. Reapplying stashed changes..."
+      if git stash pop; then
+        echo "==> Stash reapplied cleanly. Re-checking sync state..."
+        exec bash "$0" "$@"
+      else
+        echo "ERROR: 'git stash pop' did not complete successfully."
+        echo
+        if git ls-files -u | grep -q .; then
+          echo "This is a MERGE CONFLICT — your stashed changes were partially"
+          echo "applied and conflict markers (<<<<<<< / ======= / >>>>>>>) are now"
+          echo "in the working tree. The stash itself is still preserved as a backup."
+          echo
+          echo "Remediation options:"
+          echo "  A) Resolve the conflict by hand, then commit and push:"
+          echo "       cd $DIR"
+          echo "       git status                       # see conflicted paths"
+          echo "       git diff                          # inspect the markers"
+          echo "       \$EDITOR <conflicted-file>          # remove markers, keep correct content"
+          echo "       git add <conflicted-file>"
+          echo "       git commit -m 'resolve git-guard stash-pop conflict'"
+          echo "       git push origin main"
+          echo "       git stash list                    # confirm which entry is: $STASH_MSG"
+          echo "       git stash drop stash@{N}           # once confirmed no longer needed"
+          echo "  B) Abandon this merge attempt and fall back to a clean, resynced"
+          echo "     tree, then re-apply the change manually with full visibility:"
+          echo "       cd $DIR"
+          echo "       git checkout -- ."
+          echo "       git reset --hard origin/main      # now matches origin, no conflict"
+          echo "       git stash list                    # find: $STASH_MSG"
+          echo "       git stash show -p stash@{N}         # review the content"
+          echo "       git stash apply stash@{N}          # 'apply' keeps the backup; use 'pop' to also drop it"
+        else
+          echo "This looks like an UNTRACKED-FILE COLLISION, not a merge conflict"
+          echo "(a file added upstream shares a path with an untracked file in your"
+          echo "stash). No conflict markers were written; the stash was NOT applied"
+          echo "and remains fully intact."
+          echo
+          echo "Remediation options:"
+          echo "  A) Move the colliding upstream file aside, pop, then reconcile:"
+          echo "       cd $DIR"
+          echo "       git status --short                         # identify the colliding path"
+          echo "       mv <path> <path>.upstream-\$(date +%s)"
+          echo "       git stash pop"
+          echo "       diff <path> <path>.upstream-*               # reconcile manually, then remove the .upstream-* backup"
+          echo "  B) Inspect the stash without applying, and hand-merge the needed"
+          echo "     pieces instead:"
+          echo "       git stash list                              # find N"
+          echo "       git stash show -p stash@{N}"
+        fi
+        echo
+        echo "Your stash reference for this run: $STASH_MSG"
+        exit 1
+      fi
+    else
+      echo "ERROR: could not safely resync with origin/main."
+      echo "Your uncommitted changes are preserved in the stash: $STASH_MSG"
+      echo
+      echo "Remediation options:"
+      echo "  A) Follow the manual recovery steps printed above (from the"
+      echo "     behind/ahead/diverged case), THEN reapply your change:"
+      echo "       cd $DIR"
+      echo "       git stash list                 # find: $STASH_MSG"
+      echo "       git stash apply stash@{N}      # or 'pop' to also drop it once resynced"
+      echo "  B) If the stashed change is no longer needed (e.g. it's already"
+      echo "     represented in a since-merged PR), verify then drop it:"
+      echo "       git stash show -p stash@{N}"
+      echo "       git stash drop stash@{N}"
+      exit 1
+    fi
+  fi
+
   if [ "$INTERACTIVE" -eq 1 ]; then
     read -rp "Commit and push these changes to origin/main now? [y/N] " ans
   else
@@ -54,7 +298,8 @@ if [ "$DIRTY" -eq 1 ]; then
 
     if git diff --cached | grep -Eiq "$SECRET_PATTERN"; then
       echo "ERROR: possible secret detected in staged changes. Refusing to auto-commit."
-      echo "Review manually: git diff --cached"
+      echo "Review manually:"
+      echo "  cd $DIR && git diff --cached"
       git reset
       exit 1
     fi
@@ -63,80 +308,25 @@ if [ "$DIRTY" -eq 1 ]; then
 
     if git push origin main; then
       echo "==> Pushed. Re-checking sync state..."
-      exec "$0" "$@"
+      exec bash "$0" "$@"
     else
       echo "ERROR: push failed (likely diverged from origin). Aborting deploy."
-      echo "Run: cd $DIR && git status"
+      echo "Run:"
+      echo "  cd $DIR && git status"
       exit 1
     fi
   else
-    echo "Aborting deploy - commit or stash changes manually, then retry."
+    echo "Aborting deploy - commit or stash changes manually, then retry:"
+    echo "  cd $DIR"
+    echo "  git add -A && git commit -m 'your message' && git push origin main"
+    echo "  # or: git stash push -u -m 'manual-stash'"
     exit 1
   fi
 fi
 
-# ---- Case: fully in sync ----
-if [ "$LOCAL" = "$REMOTE" ]; then
-  echo "==> In sync with origin/main ($LOCAL). OK to deploy."
+# ---- Clean tree: resync with origin using the shared logic above ----
+if resync_with_origin "$LOCAL" "$REMOTE" "$BASE"; then
   exit 0
+else
+  exit 1
 fi
-
-# ---- Case: behind only (fast-forwardable) ----
-if [ "$LOCAL" = "$BASE" ]; then
-  echo "!! Local main is behind origin/main."
-  if [ "$INTERACTIVE" -eq 1 ]; then
-    read -rp "Fast-forward pull now? [y/N] " ans
-  else
-    ans="y"
-    echo "(non-interactive session — auto fast-forwarding)"
-  fi
-  if [[ "$ans" =~ ^[Yy]$ ]]; then
-    git pull --ff-only origin main
-    echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
-    exit 0
-  else
-    echo "Aborting deploy - pull manually, then retry."
-    exit 1
-  fi
-fi
-
-# ---- Case: ahead only (local commits not yet pushed) ----
-if [ "$REMOTE" = "$BASE" ]; then
-  echo "!! Local main is AHEAD of origin/main (unpushed commits):"
-  git log --oneline "origin/main..main"
-  echo
-  if [ "$INTERACTIVE" -eq 1 ]; then
-    read -rp "Push local commits to origin/main now? [y/N] " ans
-  else
-    ans="n"
-    echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
-  fi
-  if [[ "$ans" =~ ^[Yy]$ ]]; then
-    git push origin main
-    echo "==> Pushed. OK to deploy."
-    exit 0
-  else
-    echo "Aborting deploy. Review with: git log origin/main..main"
-    exit 1
-  fi
-fi
-
-# ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
-echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
-echo "!! DIVERGED: local main and origin/main have both moved independently."
-echo "!!"
-echo "!!   Local-only commits:"
-git log --oneline "$BASE..main" | sed 's/^/!!     /'
-echo "!!"
-echo "!!   Remote-only commits:"
-git log --oneline "$BASE..origin/main" | sed 's/^/!!     /'
-echo "!!"
-echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
-echo "!! Recommended recovery:"
-echo "!!   1. tar backup:      tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
-echo "!!   2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
-echo "!!   3. stash all state: git stash push -u -m 'pre-reset-snapshot'"
-echo "!!   4. reset to origin: git reset --hard origin/main"
-echo "!!   5. selectively restore needed files from the stash/backup branch"
-echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
-exit 1

deploy/provision-stack.py

@@ -0,0 +1,198 @@
+#!/usr/bin/env python3
+"""provision-stack.py — manifest-driven env-file rendering + Docker secret
+provisioning for one stack.
+
+Usage (from the CI workspace root, inside the provision-secrets step):
+    python3 deploy/provision-stack.py <stack>
+
+Reads secrets/secrets-map.yaml (data only — no code, no values) and, for the
+named stack:
+
+  1. env_template -> renders the COMPLETE env file. Placeholders of the form
+     dollar-brace VARNAME are resolved from this process's environment (the
+     Woodpecker from_secret-backed vars). The whole file is rendered every
+     run; nothing is line-edited in place, so keys can never silently go
+     missing (root cause of the 2026-09-03 ai.env incident).
+  2. env_dest -> ships the rendered file to the Swarm manager over ssh stdin
+     (write-to-temp + atomic mv, mode 600). The rendered file never touches
+     the CI workspace disk under the repo (no chance of being committed) and
+     never appears on a command line.
+  3. docker_secrets -> for each swarm-secret-name -> ENV_VAR mapping, creates
+     or rotates the Docker secret. Values are passed via ssh stdin only.
+     Rotation uses the same sha256-checksum-label convention as
+     deploy/create-secrets.sh (unchanged secrets are skipped silently).
+
+Safety properties:
+  - FAILS HARD (non-zero) if any referenced env var is missing or empty, and
+    lists the missing NAMES. A partial/broken render can never ship.
+  - FAILS HARD if any unresolved placeholder remains after rendering.
+  - NEVER prints a secret value — names and counts only.
+  - Requires SWARM_MANAGER_IP in the environment and a usable ssh identity
+    (both already set up by the provision-secrets step).
+
+Stacks not present in the manifest exit 0 with a notice, so this script is
+safe to call unconditionally; legacy case-entries in deploy.yml keep handling
+unmigrated stacks.
+"""
+
+import hashlib
+import os
+import re
+import subprocess
+import sys
+
+REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+MANIFEST_PATH = os.path.join(REPO_ROOT, "secrets", "secrets-map.yaml")
+REMOTE_BASE = "/volume1/docker/compose-files"
+PLACEHOLDER_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
+
+SSH_OPTS = ["-o", "StrictHostKeyChecking=no"]
+
+
+def die(msg: str) -> None:
+    print(f"ERROR: {msg}", file=sys.stderr)
+    sys.exit(1)
+
+
+def load_manifest() -> dict:
+    try:
+        import yaml  # py3-yaml, installed by the provision-secrets step
+    except ImportError:
+        die("PyYAML not available — provision-secrets step must apk add py3-yaml")
+    if not os.path.isfile(MANIFEST_PATH):
+        die(f"manifest not found: {MANIFEST_PATH}")
+    with open(MANIFEST_PATH, "r", encoding="utf-8") as fh:
+        data = yaml.safe_load(fh) or {}
+    stacks = data.get("stacks")
+    if not isinstance(stacks, dict):
+        die("manifest has no 'stacks:' mapping")
+    return stacks
+
+
+def ssh_target() -> str:
+    ip = os.environ.get("SWARM_MANAGER_IP", "").strip()
+    if not ip:
+        die("SWARM_MANAGER_IP is empty — check Woodpecker repo secrets")
+    return f"root@{ip}"
+
+
+def ssh_run(target: str, remote_cmd: str, stdin_data: bytes | None = None,
+            check: bool = True) -> subprocess.CompletedProcess:
+    proc = subprocess.run(
+        ["ssh", *SSH_OPTS, target, remote_cmd],
+        input=stdin_data, capture_output=True,
+    )
+    if check and proc.returncode != 0:
+        # stderr may be verbose but must never contain our secret values —
+        # we only ever send values via stdin, never embed them in remote_cmd.
+        die(f"remote command failed (rc={proc.returncode}): {remote_cmd}\n"
+            f"{proc.stderr.decode(errors='replace').strip()}")
+    return proc
+
+
+def render_template(template_path: str) -> str:
+    if not os.path.isfile(template_path):
+        die(f"env_template not found: {template_path}")
+    with open(template_path, "r", encoding="utf-8") as fh:
+        raw = fh.read()
+
+    referenced = sorted(set(PLACEHOLDER_RE.findall(raw)))
+    missing = [v for v in referenced
+               if not os.environ.get(v, "").strip()]
+    if missing:
+        die("template references vars that are MISSING or EMPTY in the CI "
+            "environment (add them via from_secret in "
+            ".woodpecker/deploy.yml provision-secrets, and as Woodpecker "
+            "secrets):\n  " + "\n  ".join(missing))
+
+    rendered = PLACEHOLDER_RE.sub(lambda m: os.environ[m.group(1)], raw)
+
+    # Belt-and-braces: nothing placeholder-shaped may survive the render.
+    leftover = sorted(set(PLACEHOLDER_RE.findall(rendered)))
+    if leftover:
+        die("unresolved placeholders remain after rendering: "
+            + ", ".join(leftover))
+
+    print(f"  [render] {template_path}: {len(referenced)} secret placeholder(s) "
+          f"resolved: {', '.join(referenced)}")
+    return rendered
+
+
+def ship_env_file(target: str, rendered: str, dest_rel: str) -> None:
+    dest = f"{REMOTE_BASE}/{dest_rel}"
+    tmp = f"{dest}.provision-tmp"
+    # Value travels over ssh stdin; never on a command line; atomic mv.
+    ssh_run(target,
+            f"umask 077 && cat > {tmp} && chmod 600 {tmp} && mv {tmp} {dest}",
+            stdin_data=rendered.encode())
+    keys = [ln.split("=", 1)[0] for ln in rendered.splitlines()
+            if "=" in ln and not ln.lstrip().startswith("#") and ln.strip()]
+    print(f"  [env] shipped {dest} ({len(keys)} keys): {', '.join(keys)}")
+
+
+def provision_docker_secret(target: str, name: str, env_var: str) -> None:
+    value = os.environ.get(env_var, "")
+    if not value.strip():
+        die(f"docker secret '{name}': env var {env_var} is missing/empty")
+
+    new_hash = hashlib.sha256(value.encode()).hexdigest()
+    probe = ssh_run(
+        target,
+        f"docker secret inspect {name} "
+        "--format '{{index .Spec.Labels \"checksum\"}}' 2>/dev/null || true",
+        check=True)
+    old_hash = probe.stdout.decode().strip()
+
+    if old_hash == new_hash:
+        print(f"  [skip]   docker secret {name} (unchanged)")
+        return
+
+    if old_hash:
+        rm = ssh_run(target, f"docker secret rm {name}", check=False)
+        if rm.returncode != 0:
+            die(f"docker secret {name}: value changed but removal failed — "
+                "it is probably referenced by a running service. Provision "
+                "under a versioned name (see vaultwarden_database_url_v2 "
+                "precedent) or scale the service down first.")
+        action = "update"
+    else:
+        action = "create"
+
+    ssh_run(target,
+            f"docker secret create --label checksum={new_hash} "
+            f"--label managed-by=woodpecker {name} -",
+            stdin_data=value.encode())
+    print(f"  [{action}] docker secret {name} (value via stdin)")
+
+
+def main() -> None:
+    if len(sys.argv) != 2:
+        die("usage: provision-stack.py <stack>")
+    stack = sys.argv[1]
+
+    stacks = load_manifest()
+    cfg = stacks.get(stack)
+    if cfg is None:
+        print(f"  [info] stack '{stack}' not in secrets-map.yaml — "
+              "legacy provisioning (deploy.yml case-entry) applies. Nothing to do.")
+        return
+
+    target = ssh_target()
+    print(f"==> provision-stack: {stack}")
+
+    template_rel = cfg.get("env_template")
+    dest_rel = cfg.get("env_dest")
+    if template_rel and not dest_rel:
+        die("env_template set but env_dest missing in manifest")
+    if template_rel:
+        rendered = render_template(os.path.join(REPO_ROOT, template_rel))
+        ship_env_file(target, rendered, dest_rel)
+
+    for name, env_var in (cfg.get("docker_secrets") or {}).items():
+        provision_docker_secret(target, name, env_var)
+
+    print(f"==> provision-stack: {stack} done")
+
+
+if __name__ == "__main__":
+    main()

secrets/secrets-map.yaml

@@ -0,0 +1,38 @@
+# ─────────────────────────────────────────────────────────────────────────────
+# secrets-map.yaml — DATA-ONLY manifest for deploy/provision-stack.py
+#
+# RULES:
+#   - This file contains NO code, NO shell, NO secret values — only names.
+#   - Each stack entry declares:
+#       env_template:   repo path of the FULL env-file template (tracked).
+#                       The template is authoritative: the COMPLETE env file is
+#                       rendered from it on every provisioning run. Nothing is
+#                       line-edited in place, so keys can never silently go
+#                       missing.
+#       env_dest:       host path (relative to /volume1/docker/compose-files/)
+#                       the rendered env file is shipped to. Rendered file
+#                       exists ONLY on the host — never committed to git.
+#       docker_secrets: map of docker-swarm-secret-name -> CI ENV VAR NAME
+#                       (Pattern C). The env var must be declared via
+#                       from_secret: in .woodpecker/deploy.yml's
+#                       provision-secrets step (Woodpecker v3 requires explicit
+#                       per-secret declaration; there is no expose-all).
+#
+# ADDING A NEW SECRET (3 small steps, no shell edits):
+#   1. Add the secret value in Woodpecker UI (repo Settings -> Secrets).
+#   2. Declare it in .woodpecker/deploy.yml provision-secrets environment:
+#      block (from_secret) — mechanical two-line addition.
+#   3. Reference it here (docker_secrets:) and/or in the stack's
+#      .env.template as a dollar-brace placeholder.
+#
+# Stacks not listed here fall through to deploy.yml's legacy case-entries
+# untouched. Migration is deliberately one stack per PR.
+# ─────────────────────────────────────────────────────────────────────────────
+stacks:
+  ai:
+    env_template: ai/ai.env.template
+    env_dest: ai/ai.env
+    docker_secrets:
+      flowagent_azure_client_id: FLOWAGENT_AZURE_CLIENT_ID
+      flowagent_azure_tenant_id: FLOWAGENT_AZURE_TENANT_ID
+      flowagent_azure_client_secret: FLOWAGENT_AZURE_CLIENT_SECRET