Fix phantom verify failure: exclude secrets/ from stack detection + errexit-proof verify's stack-ps

Root cause of PR #16's pipeline failure (#398-#401): secrets/ is a tooling folder (secrets-map.yaml, *.secrets.example), not a stack, but folder-detection (cut -d/ -f1, excluding only deploy/) computed ALL_STACKS="ai secrets" the first time a commit touched secrets/. The deploy step survived only because 'secrets' coincidentally sits in the bootstrap-tier skip list; verify had no guard — `docker stack ps secrets` exited non-zero and, under errexit, the OUTPUT=$(...) assignment killed the step before the designed WARNING path could run. The ai deploy itself was fully successful (complete 21-key render, all 6 services healthy) — the failure was pure phantom. Hazard first flagged July 2026 in PR #3 (closed unmerged). Changes: - grep -v '^deploy$' -> grep -vE '^(deploy|secrets)$' in ALL five folder-detection sites (validate, provision-secrets x2, deploy, verify) - verify: `|| true` inside the stack-ps command substitution so a genuinely missing stack warns instead of aborting (comment added) - deploy step: also rsync secrets/ tooling to the host mirror unconditionally, same treatment as deploy/ - header: 2026-09-08 incident note Everything else byte-identical to main. Dollar-escape audit done.
Merge pull request 'Provisioning v2 (take 2): data-only manifest + full-file env template + generic provisioner — replaces #15 ' (#16 ) from secrets-provisioning-v3 into main
2026-09-07 22:26:43 -07:00 · 2026-09-07 22:06:03 -07:00 · 2026-09-07 21:47:04 -07:00 · 2026-09-07 21:43:57 -07:00 · 2026-09-07 21:43:05 -07:00 · 2026-09-07 21:43:04 -07:00
14 changed files with 1062 additions and 514 deletions
@@ -2,8 +2,17 @@
 *.swp
 *.swo
 .DS_Store
 # Secrets / stack env files - never commit these.
 # deploy/global.env and immich.env-example etc. are the intended committed
 # templates/examples; actual secret-bearing .env files must stay untracked.
 .env
 .env.local
 *.env
 !*.env-example
 !deploy/global.env
 *.env.bak
 *.env.backup
 secrets/
 keys/
 *.key
@@ -15,3 +24,7 @@ keys/
 # Local overrides
 docker-compose.override.yml
 local/
 # Editor/adhoc backups that sometimes get left in the tree
 *.bak
 *.bak-*
@@ -48,6 +48,28 @@ when:
 #   already unconditionally rsynced at the top of the deploy step regardless
 #   of which stacks changed, so it's safe to exclude it from the stack list
 #   everywhere folders are detected below.
 #
 #   2026-09-03/07 INCIDENT + REDESIGN: hand-maintained grep -v + printf
 #   line-surgery cases repeatedly drifted (missing '=', duplicated keys,
 #   mismatched env var names rendering EMPTY secrets) and broke live
 #   services. Root cause: no authoritative key list and shell heredocs
 #   hostile to hand-editing. Stacks migrate one at a time to a data-driven
 #   model: secrets/secrets-map.yaml (data only) + per-stack .env.template
 #   (authoritative FULL file) + deploy/provision-stack.py (whole-file
 #   render, hard failure naming any missing value). Migrated stacks call
 #   the script; unmigrated stacks keep legacy case entries until their own
 #   PR. See the "Secrets & Deployment Architecture — Global Direction" note.
 #
 #   2026-09-08 FIX: secrets/ is a tooling/docs folder (secrets-map.yaml +
 #   *.secrets.example), not a stack — but folder-detection treated it as one
 #   the first time a commit touched it (PR #16). deploy survived only because
 #   'secrets' sits in the bootstrap-tier skip list; verify had no guard and
 #   died on `docker stack ps secrets` failing under errexit (assignment from
 #   a failing command substitution aborts the step). Fixed by excluding
 #   secrets/ alongside deploy/ in ALL folder-detection sites, and by
 #   tolerating a failing stack-ps in verify (|| true) so a genuinely missing
 #   stack produces the designed WARNING instead of killing the step. This
 #   hazard was first flagged in July (PR #3, closed unmerged).
 # ─────────────────────────────────────────────────────────────────────────────
 steps:
@@ -65,9 +87,9 @@ steps:
        FLAT=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.ya?ml$' || true)
        # Folder: any file under a subfolder (e.g. immich/immich.yml).
-        # Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and deploy/
+        # Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and the
-        # (shared tooling, not a stack — see note above).
+        # non-stack tooling folders deploy/ and secrets/ (see notes above).
-        FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
        [ -z "$FLAT" ] && [ -z "$FOLDERS" ] && echo "No stacks changed" && exit 0
@@ -163,6 +185,11 @@ steps:
        from_secret: entertainment_sparky_encryption_key
      ENTERTAINMENT_BETTER_AUTH_SECRET:
        from_secret: entertainment_better_auth_secret
      # ── ai stack (manifest-driven — secrets/secrets-map.yaml +
      #    ai/ai.env.template + deploy/provision-stack.py). Env var names
      #    below match the template placeholders EXACTLY; this block is the
      #    only per-secret touchpoint left in this file for migrated stacks
      #    (Woodpecker v3 requires explicit from_secret declarations). ──
      AI_AWS_ACCESS_KEY_ID:
        from_secret: ai_aws_access_key_id
      AI_AWS_SECRET_ACCESS_KEY:
@@ -171,16 +198,26 @@ steps:
        from_secret: ai_litellm_master_key
      AI_LITELLM_SALT_KEY:
        from_secret: ai_litellm_salt_key
-      AI_LITELLM_DB_PASSWORD:
+      AI_LITELLM_DATABASE_URL:
-        from_secret: ai_litellm_db_password
+        from_secret: ai_litellm_database_url
-      AI_WEBUI_SECRET_KEY:
+      AI_LITELLM_POSTGRES_PASSWORD:
        from_secret: ai_litellm_postgres_password
      AI_OPEN_WEBUI_SECRET_KEY:
        from_secret: ai_webui_secret_key
      AI_OPEN_WEBUI_DATABASE_URL:
        from_secret: ai_open_webui_database_url
-      AI_OAUTH_CLIENT_SECRET:
+      AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET:
        from_secret: ai_oauth_client_secret
      AI_MCPO_API_KEY:
        from_secret: ai_mcpo_api_key
      FLOWAGENT_AZURE_CLIENT_ID:
        from_secret: flowagent_azure_client_id
      FLOWAGENT_AZURE_TENANT_ID:
        from_secret: flowagent_azure_tenant_id
      FLOWAGENT_AZURE_CLIENT_SECRET:
        from_secret: flowagent_azure_client_secret
    commands:
-      - apk add --no-cache openssh-client
+      - apk add --no-cache openssh-client python3 py3-yaml
      - mkdir -p ~/.ssh
      - echo "$SSH_KEY" | base64 -d > ~/.ssh/id_rsa
      - chmod 600 ~/.ssh/id_rsa
@@ -193,14 +230,14 @@ steps:
      - |
        CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
        FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
        ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
        [ -z "$ALL_STACKS" ] && echo "No stacks changed, skipping" && exit 0
      - scp -o StrictHostKeyChecking=no deploy/create-secrets.sh root@$${SWARM_MANAGER_IP}:/tmp/cs.sh
      - |
        CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
        FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
        ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
        for STACK in $ALL_STACKS; do
@@ -221,10 +258,38 @@ steps:
              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
                create_or_update_secret 'gamma_auth_token' '$${GAMMA_AUTH_TOKEN}'";;
            git)
-              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
+              # PATTERN B, DELIBERATE (see decision notes) — git hosts the source
-                create_or_update_secret 'git_db_password'       '$${GIT_DB_PASSWORD}'
+              # of truth for every other stack's compose files, so it must be
-                create_or_update_secret 'git_runner_token'      '$${GIT_RUNNER_TOKEN}'
+              # restorable from a flat git.yaml + git.env backup alone, with zero
-                create_or_update_secret 'git_mcp_access_token'  '$${GIT_MCP_ACCESS_TOKEN}'";;
+              # dependency on a running Swarm's Docker secret store. Native Docker
              # secrets (Pattern C) can't satisfy that: they only exist inside an
              # already-running Swarm, which is exactly the circular dependency
              # this stack can't have. Mirrors the retired ai) case's grep -v +
              # printf rewrite-in-place approach, never sed (values may contain
              # slash, dollar sign, ampersand).
              #
              # TEST PHASE: target is git.env.pipelinetest, NOT the real git.env.
              # The real file is never opened for writing by this step. First run
              # seeds the test file from the real git.env (carries over all
              # non-secret lines untouched); every push after that only refreshes
              # the 3 secret lines below. Also strips the legacy GITEA_ACCESS_TOKEN
              # key name so the test file converges on the git.env.example-
              # documented GITEA_MCP_ACCESS_TOKEN key. Cutover to the real file —
              # and pointing git.yaml/stack-deploy at it — is a deliberate,
              # separate follow-up after manually diffing this render.
              # (Candidate for the secrets-map.yaml/provision-stack.py migration
              # in its own PR; kept legacy for now.)
              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/git.env.pipelinetest
                TMP=\$FILE.tmp.\$\$
                [ -f \$FILE ] || cp /volume1/docker/compose-files/git.env \$FILE
                grep -vE '^(GITEA__database__PASSWD|GITEA_RUNNER_REGISTRATION_TOKEN|GITEA_MCP_ACCESS_TOKEN|GITEA_ACCESS_TOKEN)=' \$FILE > \$TMP 2>/dev/null || touch \$TMP
                { cat \$TMP
                  printf 'GITEA__database__PASSWD=%s\n' '$${GIT_DB_PASSWORD}'
                  printf 'GITEA_RUNNER_REGISTRATION_TOKEN=%s\n' '$${GIT_RUNNER_TOKEN}'
                  printf 'GITEA_MCP_ACCESS_TOKEN=%s\n' '$${GIT_MCP_ACCESS_TOKEN}'
                } > \$FILE
                rm -f \$TMP
                echo '  [OK] git.env.pipelinetest updated - real git.env untouched'";;
            homeassistant)
              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
                create_or_update_secret 'homeassistant_frigate_rtsp_password' '$${HOMEASSISTANT_FRIGATE_RTSP_PASSWORD}'
@@ -261,40 +326,15 @@ steps:
                create_or_update_secret 'vaultwarden_admin_token'     '$${VAULTWARDEN_ADMIN_TOKEN}'
                create_or_update_secret 'vaultwarden_database_url_v2' '$${VAULTWARDEN_DATABASE_URL}'";;
            ai)
-              # NOTE: ai stack uses Pattern B (host .env, not native Docker secrets) —
+              # MIGRATED (2026-09-07) to manifest-driven provisioning after the
-              # LiteLLM/boto3 and Open WebUI don't support the _FILE convention for these
+              # line-surgery approach repeatedly drifted (dropped keys 2026-09-03;
-              # vars. Instead of Docker secrets, we regenerate ONLY the migrated lines in
+              # duplicate AI_LITELLM_POSTGRES_PASSWORD + stray legacy keys +
-              # the remote ai/ai.env in place via grep -v + printf (never sed, since values
+              # WEB_UI/WEBUI env-name mismatch rendering an EMPTY OAuth client
-              # may contain slash, dollar sign, ampersand). All other lines — including
+              # secret, found 2026-09-07). All logic lives in
-              # MCPO_API_KEY, OAUTH_CLIENT_ID, WEBUI_URL, and other non-secret config — are
+              # deploy/provision-stack.py; the authoritative key list lives in
-              # left completely untouched. MCPO_API_KEY migration is deferred to a
+              # ai/ai.env.template; the mapping lives in secrets/secrets-map.yaml.
-              # follow-up; this step never reads or writes it.
+              # This case is intentionally one line.
-              #
+              python3 deploy/provision-stack.py ai;;
              # IMPORTANT: ai.yaml's litellm service references ${AI_AWS_ACCESS_KEY_ID} /
              # ${AI_AWS_SECRET_ACCESS_KEY} (AI_-prefixed) and renders them into the
              # container as plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (see commit
              # 37ed671a, "Change AWS keys to use Woodpecker Secrets"). So ai.env must be
              # written with the AI_-prefixed key names, NOT the plain ones — writing
              # plain AWS_ACCESS_KEY_ID here would leave ${AI_AWS_ACCESS_KEY_ID}
              # unresolved at compose-render time (empty), silently breaking Bedrock auth.
              # All other migrated vars in ai.yaml use plain (unprefixed) names, so only
              # these two lines need the AI_ prefix.
              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/ai/ai.env
                TMP=\$FILE.tmp.\$\$
                grep -vE '^(AI_AWS_ACCESS_KEY_ID|AI_AWS_SECRET_ACCESS_KEY|LITELLM_MASTER_KEY|LITELLM_SALT_KEY|POSTGRES_PASSWORD|DATABASE_URL|WEBUI_SECRET_KEY|OPEN_WEBUI_DATABASE_URL|OAUTH_CLIENT_SECRET)=' \$FILE > \$TMP 2>/dev/null || touch \$TMP
                { cat \$TMP
                  printf 'AI_AWS_ACCESS_KEY_ID=%s\n' '$${AI_AWS_ACCESS_KEY_ID}'
                  printf 'AI_AWS_SECRET_ACCESS_KEY=%s\n' '$${AI_AWS_SECRET_ACCESS_KEY}'
                  printf 'LITELLM_MASTER_KEY=%s\n' '$${AI_LITELLM_MASTER_KEY}'
                  printf 'LITELLM_SALT_KEY=%s\n' '$${AI_LITELLM_SALT_KEY}'
                  printf 'POSTGRES_PASSWORD=%s\n' '$${AI_LITELLM_DB_PASSWORD}'
                  printf 'DATABASE_URL=postgresql://LiteLLM:%s@postgresql:5432/litellm\n' '$${AI_LITELLM_DB_PASSWORD}'
                  printf 'WEBUI_SECRET_KEY=%s\n' '$${AI_WEBUI_SECRET_KEY}'
                  printf 'OPEN_WEBUI_DATABASE_URL=%s\n' '$${AI_OPEN_WEBUI_DATABASE_URL}'
                  printf 'OAUTH_CLIENT_SECRET=%s\n' '$${AI_OAUTH_CLIENT_SECRET}'
                } > \$FILE
                rm -f \$TMP
                echo '  [OK] ai/ai.env secrets updated'";;
            entertainment)
              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
                create_or_update_secret 'entertainment_discord_token'          '$${ENTERTAINMENT_DISCORD_TOKEN}'
@@ -331,7 +371,7 @@ steps:
      - |
        CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
        FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
        ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
        [ -z "$ALL_STACKS" ] && echo "No stacks changed" && exit 0
@@ -339,6 +379,13 @@ steps:
        rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
          deploy/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/deploy/
        # Sync secrets/ tooling (manifest + examples) alongside deploy/ —
        # provision-stack.py reads secrets/secrets-map.yaml from the CI
        # checkout, but the host mirror should stay complete for emergency
        # manual provisioning runs.
        rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
          secrets/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/secrets/
        for STACK in $ALL_STACKS; do
          echo "--- Deploying: $STACK ---"
          # Sync files to host first (always, even for bootstrap stacks)
@@ -352,7 +399,7 @@ steps:
          # Bootstrap-tier guard: file synced to host, deploy is MANUAL
          case "$STACK" in
-            traefik|woodpecker|postgresql|secrets)
+            traefik|woodpecker|postgresql|secrets|git)
              echo "  [BOOTSTRAP] $STACK: file synced. Deploy is MANUAL."
              echo "  Run: bash /volume1/docker/compose-files/deploy/stack-deploy.sh $STACK"
              continue ;;
@@ -385,7 +432,7 @@ steps:
      - |
        CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
        FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
-        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
+        FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
        ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
        [ -z "$ALL_STACKS" ] && exit 0
@@ -396,6 +443,10 @@ steps:
        # "nothing found in stack" output on ordinary deploys (e.g. vaultwarden,
        # 2026-08-25). Retry with backoff instead of a single fixed sleep, and
        # only warn (don't fail the pipeline) if tasks never show up.
        # 2026-09-08: `|| true` inside the command substitution is REQUIRED —
        # this step runs under errexit, and an assignment from a failing
        # command substitution (e.g. `docker stack ps` on a stack that doesn't
        # exist) kills the whole step before the WARNING path can run.
        ATTEMPTS=6
        DELAY=5
        for STACK in $ALL_STACKS; do
@@ -404,7 +455,7 @@ steps:
          while [ "$i" -le "$ATTEMPTS" ]; do
            OUTPUT=$(ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
              "docker stack ps $STACK --filter desired-state=running \
-              --format '  {{.Name}} {{.CurrentState}}'" 2>/dev/null)
+              --format '  {{.Name}} {{.CurrentState}}'" 2>/dev/null || true)
            if [ -n "$OUTPUT" ]; then
              echo "$OUTPUT"
              break
@@ -0,0 +1,55 @@
 # ─────────────────────────────────────────────────────────────────────────────
 # ai.env.template — AUTHORITATIVE template for ai/ai.env (rendered by
 # deploy/provision-stack.py per secrets/secrets-map.yaml).
 #
 #   - This file IS the complete key list for ai.env. The whole file is
 #     rendered on every provisioning run — no line surgery, so a key can
 #     never silently go missing again (root cause of the 2026-09-03 outage).
 #   - Key names match EXACTLY what ai/ai.yaml references (AI_<SERVICE>_*
 #     naming adopted on main 2026-09-06).
 #   - Non-secret config lives here as LITERAL values (visible, reviewable).
 #   - Secret values are dollar-brace placeholders resolved from the CI env
 #     (Woodpecker from_secret vars) at provisioning time. provision-stack.py
 #     FAILS HARD if any placeholder is missing/empty.
 #   - The rendered ai/ai.env exists only on the host (gitignored).
 #   - Rendered by provision-stack.py, NOT Woodpecker's yaml preprocessor —
 #     single-dollar placeholders are safe here (deploy.yml's double-dollar
 #     rule does NOT apply to this file).
 #
 # Consumed by ai/ai.yaml. DOMAIN_NAME comes from deploy/global.env, not here.
 # ─────────────────────────────────────────────────────────────────────────────
 # ── LiteLLM (non-secret config) ──────────────────────────────────────────────
 AI_AWS_REGION_NAME=us-east-2
 AI_LITELLM_MODIFY_PARAMS=False
 AI_LITELLM_DATABASE_MIGRATIONS=True
 # ── LiteLLM (secrets) ────────────────────────────────────────────────────────
 AI_AWS_ACCESS_KEY_ID=${AI_AWS_ACCESS_KEY_ID}
 AI_AWS_SECRET_ACCESS_KEY=${AI_AWS_SECRET_ACCESS_KEY}
 AI_LITELLM_MASTER_KEY=${AI_LITELLM_MASTER_KEY}
 AI_LITELLM_SALT_KEY=${AI_LITELLM_SALT_KEY}
 # Full connection URL is itself a secret (ai_litellm_database_url) — the
 # URL structure never appears in git.
 AI_LITELLM_DATABASE_URL=${AI_LITELLM_DATABASE_URL}
 AI_LITELLM_POSTGRES_PASSWORD=${AI_LITELLM_POSTGRES_PASSWORD}
 # ── Open WebUI (non-secret config) ───────────────────────────────────────────
 AI_OPEN_WEBUI_URL=https://ai.bryanmail.net
 AI_OPEN_WEBUI_ENABLE_OAUTH_SIGNUP=true
 AI_OPEN_WEBUI_OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
 AI_OPEN_WEBUI_OAUTH_PROVIDER_NAME=Authentik
 AI_OPEN_WEBUI_OPENID_PROVIDER_URL=https://auth.bryanmail.net/application/o/open-web-ui/.well-known/openid-configuration
 # OAuth client ID is a public identifier by OAuth2 design (it is sent to the
 # browser); the client SECRET below is the protected credential.
 AI_OPEN_WEBUI_OAUTH_CLIENT_ID=hVmhi1dS3TnG2cUw5QwLOx5FDLSWtnQUdZyeB5zK
 AI_OPEN_WEBUI_OAUTH_SCOPES=openid email profile
 AI_OPEN_WEBUI_OPENID_REDIRECT_URI=https://ai.bryanmail.net/oauth/oidc/callback
 # ── Open WebUI (secrets) ─────────────────────────────────────────────────────
 AI_OPEN_WEBUI_SECRET_KEY=${AI_OPEN_WEBUI_SECRET_KEY}
 AI_OPEN_WEBUI_DATABASE_URL=${AI_OPEN_WEBUI_DATABASE_URL}
 AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET=${AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET}
 # ── mcpo / mcpo-critical (secrets) ───────────────────────────────────────────
 MCPO_API_KEY=${AI_MCPO_API_KEY}
@@ -1,11 +1,6 @@
 # NOTE: litellm's AWS/DB/master-key secrets, open-webui's secret key/DB URL,
 # and the OAuth client secret are provisioned into ai/ai.env at deploy time
 # from Woodpecker secrets (see PRs #4, #6, #7) -- not committed here.
 # MCPO_API_KEY (used by mcpo and mcpo-critical) is NOT yet migrated; it
 # remains manually managed in ai/ai.env by design (see PR #4 discussion).
 services:
  open-webui:
-    image: ghcr.io/open-webui/open-webui:0.11.1
+    image: ghcr.io/open-webui/open-webui:0.11.3 #
    entrypoint:
    - /bin/bash
    - /app/tools/startup.sh
@@ -21,20 +16,20 @@ services:
      start_period: 60s
    environment:
    - OLLAMA_BASE_URL=http://ollama-intel-arc:11434
-    - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
+    - WEBUI_SECRET_KEY=${AI_OPEN_WEBUI_SECRET_KEY}
    - WEBUI_DB_HOST=postgresql
-    - DATABASE_URL=${OPEN_WEBUI_DATABASE_URL}
+    - DATABASE_URL=${AI_OPEN_WEBUI_DATABASE_URL}
    - ENABLE_TOOL_SERVER_CALLING=true
    - TOOL_SERVER_CALLING=true
-    - WEBUI_URL=${WEBUI_URL}
+    - WEBUI_URL=${AI_OPEN_WEBUI_URL}
-    - ENABLE_OAUTH_SIGNUP=${ENABLE_OAUTH_SIGNUP}
+    - ENABLE_OAUTH_SIGNUP=${AI_OPEN_WEBUI_ENABLE_OAUTH_SIGNUP}
-    - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=${OAUTH_MERGE_ACCOUNTS_BY_EMAIL}
+    - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=${AI_OPEN_WEBUI_OAUTH_MERGE_ACCOUNTS_BY_EMAIL}
-    - OAUTH_PROVIDER_NAME=${OAUTH_PROVIDER_NAME}
+    - OAUTH_PROVIDER_NAME=${AI_OPEN_WEBUI_OAUTH_PROVIDER_NAME}
-    - OPENID_PROVIDER_URL=${OPENID_PROVIDER_URL}
+    - OPENID_PROVIDER_URL=${AI_OPEN_WEBUI_OPENID_PROVIDER_URL}
-    - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
+    - OAUTH_CLIENT_ID=${AI_OPEN_WEBUI_OAUTH_CLIENT_ID}
-    - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
+    - OAUTH_CLIENT_SECRET=${AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET}
-    - OAUTH_SCOPES=${OAUTH_SCOPES}
+    - OAUTH_SCOPES=${AI_OPEN_WEBUI_OAUTH_SCOPES}
-    - OPENID_REDIRECT_URI=${OPENID_REDIRECT_URI}
+    - OPENID_REDIRECT_URI=${AI_OPEN_WEBUI_OPENID_REDIRECT_URI}
    networks:
    - traefik_backend
    - postgresql_db-backend
@@ -67,13 +62,13 @@ services:
    environment:
    - AWS_ACCESS_KEY_ID=${AI_AWS_ACCESS_KEY_ID}
    - AWS_SECRET_ACCESS_KEY=${AI_AWS_SECRET_ACCESS_KEY}
-    - AWS_REGION_NAME=${AWS_REGION_NAME}
+    - AWS_REGION_NAME=${AI_AWS_REGION_NAME}
-    - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
+    - LITELLM_MASTER_KEY=${AI_LITELLM_MASTER_KEY}
-    - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
+    - LITELLM_SALT_KEY=${AI_LITELLM_SALT_KEY}
-    - DATABASE_URL=${DATABASE_URL}
+    - DATABASE_URL=${AI_LITELLM_DATABASE_URL}
-    - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+    - POSTGRES_PASSWORD=${AI_LITELLM_POSTGRES_PASSWORD}
-    - LITELLM_MODIFY_PARAMS=${LITELLM_MODIFY_PARAMS}
+    - LITELLM_MODIFY_PARAMS=${AI_LITELLM_MODIFY_PARAMS}
-    - LITELLM_DATABASE_MIGRATIONS=${LITELLM_DATABASE_MIGRATIONS}
+    - LITELLM_DATABASE_MIGRATIONS=${AI_LITELLM_DATABASE_MIGRATIONS}
    networks:
    - traefik_backend
    - postgresql_db-backend
@@ -132,7 +127,7 @@ services:
      labels:
      - traefik.enable=false
  mcpo:
-    image: ghcr.io/open-webui/mcpo:main
+    image: git.bryanmail.net/homelab/flowagent-mcpo:c5b22618404a
    command:
    - --config
    - /app/config/config.json
@@ -146,6 +141,13 @@ services:
    - /volume1/docker/mcpo/data:/mcpo_data
    - /volume1/docker/cronicle/ssh_keys:/app/ssh_keys:ro
    - /volume1/docker/mcpo/uv-cache:/app/uv-cache
    secrets:
    - source: flowagent_azure_client_id
      target: flowagent_azure_client_id
    - source: flowagent_azure_tenant_id
      target: flowagent_azure_tenant_id
    - source: flowagent_azure_client_secret
      target: flowagent_azure_client_secret
    networks:
    - traefik_backend
    deploy:
@@ -178,6 +180,13 @@ services:
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=forwardAuth-authentik@file, crowdsec@file
      - traefik.swarm.network=traefik_backend
 secrets:
  flowagent_azure_client_id:
    external: true
  flowagent_azure_tenant_id:
    external: true
  flowagent_azure_client_secret:
    external: true
 networks:
  traefik_backend:
    external: true
@@ -22,7 +22,7 @@ def merge_envs(base_path, override_path):
    merged = {**base, **override}
    return list(merged.items())
-# ─────────────────────────────────────────────────────────────────────────────
+# ──────────────────────────────────────────────────────────────────────────
 # 2026-08-26 FIX — double-interpolation truncation bug (Pattern B stacks):
 #
 #   stack-deploy.sh's single-file/no-extras path is:
@@ -57,15 +57,179 @@ def merge_envs(base_path, override_path):
 #   one level of escaping ('$$' -> literal '$'), landing on the correct
 #   original single '$' with no leftover variable-reference lookalike.
 #
-#   Only applied in export/export_merged (which feed `eval` to set the
+#   ONLY applies to the single-file path (no `docker compose config` step
-#   actual values envsubst reads) — NOT in vars/vars_merged, which just
+#   downstream). See export_raw/export_raw_merged below for why the
-#   build envsubst's space-separated $VARNAME allowlist string and have
+#   folder+extras path must NOT use this.
-#   nothing to do with actual values.
+# ──────────────────────────────────────────────────────────────────────────
 # ─────────────────────────────────────────────────────────────────────────────
 def escape_dollar(v):
    return v.replace('$', '$$')
 # ──────────────────────────────────────────────────────────────────────────
 # 2026-08-26 FIX #2 — SECOND double-interpolation bug, folder+extras path
 # (discovered fixing IMMICH_KIOSK_BASICAUTH, a bcrypt hash full of '$'):
 #
 #   The folder+extras render path is:
 #     docker compose <files> config | strip | envsubst "$VARS"
 #     -> docker stack deploy -c -
 #
 #   `docker compose config` performs its OWN ${VAR} interpolation AND its
 #   own re-escaping of the output: any literal '$' character that ends up
 #   in the rendered YAML — escaped or not — gets doubled to '$$' by
 #   `docker compose config` itself, unconditionally, as part of producing
 #   spec-safe output. Confirmed by isolated test:
 #     raw MYVAR='a$b$c'   -> docker compose config -> "a$$b$$c"   (correct,
 #                                                        one level added)
 #     escaped MYVAR='a$$b$$c' (i.e. pre-doubled by escape_dollar) ->
 #       docker compose config -> "a$$$$b$$$$c"        (WRONG, doubled twice)
 #
 #   `docker stack deploy -c -` still only removes exactly ONE level of
 #   escaping on its way in (confirmed: "a$$b$$c" -> container label
 #   "a$b$c", correct). So across the whole folder+extras pipeline there is
 #   exactly ONE implicit escaping step (`docker compose config`) and ONE
 #   implicit un-escaping step (`docker stack deploy`) already built in —
 #   pre-escaping the exported value on top of that leaves one extra,
 #   uncollapsed level of '$$' in the final container label/env value.
 #
 #   Confirmed impact (2026-08-26): IMMICH_KIOSK_BASICAUTH
 #   ("BabyBryan:$2y$05$...") rendered as "BabyBryan:$$2y$$05$$..." in the
 #   final container label — Traefik basic auth would never match the real
 #   password hash, silently locking out the kiosk with no error.
 #
 #   Fix: use export_raw / export_raw_merged (NO escape_dollar) whenever the
 #   render path goes through `docker compose config` — i.e. any stack with
 #   extension files. Use export / export_merged (WITH escape_dollar) only
 #   for the single-file path, which has no `docker compose config` step and
 #   therefore only Swarm's own interpolation pass to protect against.
 # ──────────────────────────────────────────────────────────────────────────
 # ──────────────────────────────────────────────────────────────────────────
 # 2026-08-26 FIX — depends_on long-form vs Swarm short-form:
 #
 #   Folder-based stacks with extension files (e.g. immich + hwaccel.*.yml)
 #   go through the `docker compose <files> config | strip | docker stack
 #   deploy -c -` merge path. Modern `docker compose config` normalizes the
 #   short-form list syntax:
 #     depends_on: [database, redis]
 #   into the long-form condition mapping:
 #     depends_on:
 #       database:
 #         condition: service_started
 #         required: true
 #       redis:
 #         condition: service_started
 #         required: true
 #
 #   `docker stack deploy` (Swarm mode) does NOT understand the long-form
 #   mapping and rejects it with: "services.<svc>.depends_on must be a list".
 #   Single-file stacks (no extras) never hit this because they skip the
 #   `docker compose config` step entirely and go straight through envsubst.
 #
 #   Fix: collapse any long-form depends_on mapping back into the Swarm-
 #   compatible short-form list, purely as a text transform on the rendered
 #   YAML, right alongside the existing env_file stripping.
 # ──────────────────────────────────────────────────────────────────────────
 _DEP_RE = re.compile(r'^([ \t]*)depends_on:\s*$')
 _CHILD_KEY_RE = re.compile(r'^([ \t]*)([\w.\-]+):\s*$')
 _CHILD_LIST_RE = re.compile(r'^([ \t]*)-\s*(\S+)\s*$')
 def collapse_depends_on(text):
    lines = text.split('\n')
    out = []
    i = 0
    n = len(lines)
    while i < n:
        line = lines[i]
        m = _DEP_RE.match(line)
        if not m:
            out.append(line)
            i += 1
            continue
        indent = m.group(1)
        base_indent = len(indent)
        out.append(line)
        i += 1
        services = []
        parse_ok = True
        while i < n:
            l = lines[i]
            if l.strip() == '':
                i += 1
                continue
            cur_indent = len(l) - len(l.lstrip(' \t'))
            if cur_indent <= base_indent:
                break  # dedent — end of this depends_on block
            lm = _CHILD_LIST_RE.match(l)
            if lm and cur_indent == base_indent + 2:
                services.append(lm.group(2))
                i += 1
                continue
            km = _CHILD_KEY_RE.match(l)
            if km and cur_indent == base_indent + 2:
                services.append(km.group(2))
                i += 1
                # skip nested condition/required/restart lines under this service
                while i < n:
                    nl = lines[i]
                    if nl.strip() == '':
                        i += 1
                        continue
                    nl_indent = len(nl) - len(nl.lstrip(' \t'))
                    if nl_indent > base_indent + 2:
                        i += 1
                        continue
                    break
                continue
            # Unexpected shape — bail without transforming to avoid corrupting output
            parse_ok = False
            break
        if not parse_ok or not services:
            # Re-emit whatever we consumed as-is (best effort: fall back to
            # original slice) rather than risk mangling an unfamiliar shape.
            pass
        for svc in services:
            out.append(indent + '  - ' + svc)
    return '\n'.join(out)
 # ──────────────────────────────────────────────────────────────────────────
 # 2026-08-26 FIX — top-level `name:` property rejected by Swarm:
 #
 #   `docker compose config` (Compose Spec output) emits a top-level
 #   `name: <project>` key as the first line of the rendered document, e.g.:
 #     name: immich
 #     services:
 #       ...
 #
 #   This is valid Compose Spec but `docker stack deploy` (Swarm mode) uses
 #   a stricter schema that does NOT allow a top-level `name` property, and
 #   rejects the whole document with: "(root) Additional property name is
 #   not allowed". Single-file stacks (no extras) never hit this because
 #   they skip `docker compose config` and go straight through envsubst on
 #   the raw source YAML, which never had a top-level `name:` to begin with.
 #
 #   Fix: drop ONLY the top-level `name:` line (column 0, i.e. no leading
 #   whitespace) at the START of the document. Nested `name:` fields under
 #   networks/volumes/secrets (e.g. `name: postgresql_db-backend`, always
 #   indented) are legitimate and must NOT be touched.
 # ──────────────────────────────────────────────────────────────────────────
 _TOP_NAME_RE = re.compile(r'^name:\s*\S+\s*$')
 def strip_top_level_name(text):
    lines = text.split('\n')
    if lines and _TOP_NAME_RE.match(lines[0]):
        lines = lines[1:]
    return '\n'.join(lines)
 mode = sys.argv[1]
 if mode == 'export':
    for k, v in parse_env(sys.argv[2]):
@@ -74,6 +238,15 @@ elif mode == 'export_merged':
    # export_merged <global.env> <stack.env>
    for k, v in merge_envs(sys.argv[2], sys.argv[3]):
        print('export {}={}'.format(k, repr(escape_dollar(v))))
 elif mode == 'export_raw':
    # export_raw <env-file> — NO escape_dollar. Use for the docker-compose-
    # config render path (folder+extras), which does its own '$' escaping.
    for k, v in parse_env(sys.argv[2]):
        print('export {}={}'.format(k, repr(v)))
 elif mode == 'export_raw_merged':
    # export_raw_merged <global.env> <stack.env> — NO escape_dollar.
    for k, v in merge_envs(sys.argv[2], sys.argv[3]):
        print('export {}={}'.format(k, repr(v)))
 elif mode == 'vars':
    print(' '.join('$' + k for k, v in parse_env(sys.argv[2])))
 elif mode == 'vars_merged':
@@ -82,4 +255,6 @@ elif mode == 'vars_merged':
 elif mode == 'strip':
    t = sys.stdin.read()
    t = re.sub(r'[ \t]*env_file:[ \t]*\n([ \t]+-[^\n]*\n)+', '', t)
    t = collapse_depends_on(t)
    t = strip_top_level_name(t)
    sys.stdout.write(t)
@@ -0,0 +1,142 @@
 #!/usr/bin/env bash
 # git-guard.sh — Ensures the compose-files working tree is in sync with Gitea
 # before any deploy proceeds. Called automatically by stack-deploy.sh.
 #
 # Behavior:
 #   - Clean + up to date        -> pass silently
 #   - Clean + behind (ff-only)  -> auto `git pull --ff-only`, then pass
 #   - Ahead only (unpushed)     -> interactive: offer to push; non-interactive: BLOCK
 #   - Dirty tracked changes     -> offer to commit + push right now
 #                                   (auto in non-interactive/CI runs, after a
 #                                   secret-pattern scan of the staged diff)
 #   - Diverged (local AND       -> REFUSE. Never auto-resolves. Prints the
 #     remote both moved)           backup/stash/reset recovery steps and exits.
 #
 # Exit codes: 0 = safe to deploy, 1 = blocked, needs human intervention
 set -euo pipefail
 DIR="/volume1/docker/compose-files"
 cd "$DIR"
 # Non-interactive detection (Woodpecker/cron have no TTY on stdin)
 INTERACTIVE=0
 [ -t 0 ] && INTERACTIVE=1
 echo "==> git-guard: checking repo sync state"
 git fetch origin --quiet
 LOCAL="$(git rev-parse main)"
 REMOTE="$(git rev-parse origin/main)"
 BASE="$(git merge-base main origin/main)"
 DIRTY=0
 git status --porcelain | grep -q . && DIRTY=1
 SECRET_PATTERN='(-----BEGIN [A-Z]+ PRIVATE KEY-----|AKIA[0-9A-Z]{16}|xox[baprs]-[0-9a-zA-Z-]+|password[[:space:]]*[:=][[:space:]]*[^$ ]|api[_-]?key[[:space:]]*[:=][[:space:]]*[^$ ])'
 # ---- Case: dirty tracked changes ----
 if [ "$DIRTY" -eq 1 ]; then
  echo "!! WORKING TREE DIRTY — uncommitted changes detected:"
  git status --short
  echo
  if [ "$INTERACTIVE" -eq 1 ]; then
    read -rp "Commit and push these changes to origin/main now? [y/N] " ans
  else
    ans="y"
    echo "(non-interactive session — auto-committing and pushing)"
  fi
  if [[ "$ans" =~ ^[Yy]$ ]]; then
    git add -A
    if git diff --cached | grep -Eiq "$SECRET_PATTERN"; then
      echo "ERROR: possible secret detected in staged changes. Refusing to auto-commit."
      echo "Review manually: git diff --cached"
      git reset
      exit 1
    fi
    git commit -m "chore(auto): git-guard autofix - commit local changes before deploy $(date -u +%Y-%m-%dT%H:%M:%SZ)"
    if git push origin main; then
      echo "==> Pushed. Re-checking sync state..."
      exec "$0" "$@"
    else
      echo "ERROR: push failed (likely diverged from origin). Aborting deploy."
      echo "Run: cd $DIR && git status"
      exit 1
    fi
  else
    echo "Aborting deploy - commit or stash changes manually, then retry."
    exit 1
  fi
 fi
 # ---- Case: fully in sync ----
 if [ "$LOCAL" = "$REMOTE" ]; then
  echo "==> In sync with origin/main ($LOCAL). OK to deploy."
  exit 0
 fi
 # ---- Case: behind only (fast-forwardable) ----
 if [ "$LOCAL" = "$BASE" ]; then
  echo "!! Local main is behind origin/main."
  if [ "$INTERACTIVE" -eq 1 ]; then
    read -rp "Fast-forward pull now? [y/N] " ans
  else
    ans="y"
    echo "(non-interactive session — auto fast-forwarding)"
  fi
  if [[ "$ans" =~ ^[Yy]$ ]]; then
    git pull --ff-only origin main
    echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
    exit 0
  else
    echo "Aborting deploy - pull manually, then retry."
    exit 1
  fi
 fi
 # ---- Case: ahead only (local commits not yet pushed) ----
 if [ "$REMOTE" = "$BASE" ]; then
  echo "!! Local main is AHEAD of origin/main (unpushed commits):"
  git log --oneline "origin/main..main"
  echo
  if [ "$INTERACTIVE" -eq 1 ]; then
    read -rp "Push local commits to origin/main now? [y/N] " ans
  else
    ans="n"
    echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
  fi
  if [[ "$ans" =~ ^[Yy]$ ]]; then
    git push origin main
    echo "==> Pushed. OK to deploy."
    exit 0
  else
    echo "Aborting deploy. Review with: git log origin/main..main"
    exit 1
  fi
 fi
 # ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
 echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
 echo "!! DIVERGED: local main and origin/main have both moved independently."
 echo "!!"
 echo "!!   Local-only commits:"
 git log --oneline "$BASE..main" | sed 's/^/!!     /'
 echo "!!"
 echo "!!   Remote-only commits:"
 git log --oneline "$BASE..origin/main" | sed 's/^/!!     /'
 echo "!!"
 echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
 echo "!! Recommended recovery:"
 echo "!!   1. tar backup:      tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
 echo "!!   2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
 echo "!!   3. stash all state: git stash push -u -m 'pre-reset-snapshot'"
 echo "!!   4. reset to origin: git reset --hard origin/main"
 echo "!!   5. selectively restore needed files from the stash/backup branch"
 echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
 exit 1
@@ -0,0 +1,209 @@
 #!/usr/bin/env python3
 """
 mount-guard.py — Pre-flight check for bind mount paths before `docker stack deploy`.
 Usage: mount-guard.py <rendered-compose.yml>
 Reads the FINAL rendered compose YAML (after env substitution, right before
 it would be handed to `docker stack deploy -c -`) and checks:
  1. MISSING PATHS — every bind-mount source path referenced by any service
     actually exists on disk. Swarm rejects the task at schedule time if not
     (see: "bind source path does not exist"), but catching it here is
     faster and clearer, and gives the option to create it on the spot.
  2. SUSPICIOUS POSTGRES DATA DIRS — for any bind mount whose *target*
     looks like a Postgres data directory, warn if the *source* exists but
     is empty or missing a PG_VERSION file. This is the class of near-miss
     from the 2026-08-26 Immich incident: a wrong-but-existing empty path
     would have let Postgres silently initialize a brand-new database while
     the real data sat orphaned elsewhere, with no error at all.
 Exit codes: 0 = safe to deploy, 1 = blocked / aborted.
 Interactive sessions get a prompt with remediation options. Non-interactive
 sessions (Woodpecker, cron, CI) NEVER auto-proceed past a finding here —
 this is a data-safety check, not a convenience autofix.
 """
 import sys
 import os
 import re
 import yaml
 def is_bind_like(path):
    return isinstance(path, str) and (
        path.startswith('/') or path.startswith('./') or path.startswith('../')
    )
 def gather_bind_mounts(doc):
    """Return list of (service_name, target, source) for host-path bind mounts.
    Handles both compose-file syntaxes that can appear in a rendered stack:
      - short form:  "hostpath:containerpath[:mode]"        (raw single-file path)
      - long form:   {type: bind, source: ..., target: ...} (post `docker compose config`)
    Named volumes (source has no leading '/', './', '../') are intentionally skipped.
    """
    results = []
    services = doc.get('services') or {}
    for svc_name, svc in services.items():
        if not isinstance(svc, dict):
            continue
        for v in (svc.get('volumes') or []):
            if isinstance(v, str):
                parts = v.split(':')
                if len(parts) >= 2 and is_bind_like(parts[0]):
                    results.append((svc_name, parts[1], parts[0]))
            elif isinstance(v, dict):
                if v.get('type') == 'bind':
                    src = v.get('source')
                    tgt = v.get('target')
                    if is_bind_like(src):
                        results.append((svc_name, tgt, src))
        # devices: short form "host:container[:mode]" (rarely used in this repo —
        # devices are conventionally expressed as bind-mounted volumes instead,
        # for Docker Swarm / DDM compatibility — but handle it if present).
        for d in (svc.get('devices') or []):
            if isinstance(d, str):
                parts = d.split(':')
                if parts and is_bind_like(parts[0]):
                    tgt = parts[1] if len(parts) > 1 else parts[0]
                    results.append((svc_name, tgt, parts[0]))
    return results
 _PG_TARGET_RE = re.compile(r'postgres', re.IGNORECASE)
 def looks_like_postgres_target(target):
    return bool(target) and bool(_PG_TARGET_RE.search(target))
 def check_postgres_data(path):
    """Return a warning string if `path` looks like an empty/uninitialized
    Postgres data directory. Returns None if it looks fine. Caller must
    ensure `path` already exists before calling this."""
    if not os.path.isdir(path):
        # Exists but isn't a directory (e.g. a file) — a different, separate
        # problem; the missing-path style check doesn't apply, but this is
        # clearly wrong too.
        return "exists but is not a directory"
    try:
        entries = os.listdir(path)
    except PermissionError:
        return "cannot list directory contents (permission denied) — unable to verify"
    if not entries:
        return "directory exists but is EMPTY — looks like an uninitialized/wrong Postgres data dir"
    if 'PG_VERSION' not in entries:
        return "directory exists and is non-empty but has no PG_VERSION file — does not look like a real Postgres data directory"
    return None
 def prompt(question, choices):
    """choices: dict of key -> description. Returns the chosen key (lowercase)."""
    print("Options:")
    for k, desc in choices.items():
        print(f"  [{k}] {desc}")
    while True:
        ans = input(f"{question} [{'/'.join(choices.keys())}]: ").strip().lower()
        if ans in choices:
            return ans
        print(f"Please enter one of: {', '.join(choices.keys())}")
 def main():
    if len(sys.argv) != 2:
        print("Usage: mount-guard.py <rendered-compose.yml>", file=sys.stderr)
        return 1
    rendered_path = sys.argv[1]
    with open(rendered_path) as f:
        doc = yaml.safe_load(f)
    if not doc or 'services' not in doc:
        print("!! mount-guard: rendered YAML has no 'services' key — refusing to guess, blocking.")
        return 1
    mounts = gather_bind_mounts(doc)
    missing = []
    pg_warnings = []
    for svc_name, target, src in mounts:
        if not os.path.exists(src):
            missing.append((svc_name, target, src))
            continue
        if looks_like_postgres_target(target):
            warn = check_postgres_data(src)
            if warn:
                pg_warnings.append((svc_name, target, src, warn))
    if not missing and not pg_warnings:
        print(f"==> mount-guard: {len(mounts)} bind mount path(s) checked, all present and sane. OK to deploy.")
        return 0
    print("!! mount-guard found potential issues with bind mount paths:\n")
    if missing:
        print("MISSING PATHS (Swarm will reject the task if these aren't created first):")
        for svc_name, target, src in missing:
            print(f"  - service '{svc_name}': {src}  (mounted at {target})")
        print()
    if pg_warnings:
        print("SUSPICIOUS POSTGRES DATA DIRECTORIES:")
        print("  Path exists, but contents don't look like a real Postgres data dir.")
        print("  This is the exact shape of the 2026-08-26 Immich near-miss: a wrong")
        print("  bind path that HAPPENED to exist (empty) would have let Postgres")
        print("  silently init a new empty DB while the real data sat orphaned")
        print("  elsewhere — with no error or warning from Docker at all.")
        for svc_name, target, src, warn in pg_warnings:
            print(f"  - service '{svc_name}': {src}  (mounted at {target})")
            print(f"      {warn}")
        print()
    interactive = sys.stdin.isatty()
    if not interactive:
        print("(non-interactive session — mount-guard will NOT auto-proceed on any finding above)")
        print("Re-run interactively to review and decide, or fix the paths and retry.")
        return 1
    if missing:
        choices = {
            'a': 'Abort deploy (recommended if any path above is unexpected)',
            'm': 'mkdir -p the missing path(s) now, then continue',
            'c': 'Continue anyway without creating them (will likely fail at Swarm scheduling)',
        }
        ans = prompt("Missing bind mount paths found. Choice", choices)
        if ans == 'a':
            print("Aborting deploy.")
            return 1
        if ans == 'm':
            for _, _, src in missing:
                print(f"  mkdir -p {src}")
                os.makedirs(src, exist_ok=True)
            print("==> Created missing path(s). Continuing.")
        # 'c' falls through and continues without creating
    if pg_warnings:
        choices = {
            'a': 'Abort deploy (recommended unless you intended a fresh Postgres init here)',
            'c': 'Continue anyway (I have verified this is expected, e.g. legitimate first-time init)',
        }
        ans = prompt("Suspicious Postgres data directory found. Choice", choices)
        if ans == 'a':
            print("Aborting deploy.")
            return 1
        print("==> Continuing deploy despite Postgres data warning, per operator confirmation.")
    return 0
 if __name__ == '__main__':
    sys.exit(main())
@@ -0,0 +1,198 @@
 #!/usr/bin/env python3
 """provision-stack.py — manifest-driven env-file rendering + Docker secret
 provisioning for one stack.
 Usage (from the CI workspace root, inside the provision-secrets step):
    python3 deploy/provision-stack.py <stack>
 Reads secrets/secrets-map.yaml (data only — no code, no values) and, for the
 named stack:
  1. env_template -> renders the COMPLETE env file. Placeholders of the form
     dollar-brace VARNAME are resolved from this process's environment (the
     Woodpecker from_secret-backed vars). The whole file is rendered every
     run; nothing is line-edited in place, so keys can never silently go
     missing (root cause of the 2026-09-03 ai.env incident).
  2. env_dest -> ships the rendered file to the Swarm manager over ssh stdin
     (write-to-temp + atomic mv, mode 600). The rendered file never touches
     the CI workspace disk under the repo (no chance of being committed) and
     never appears on a command line.
  3. docker_secrets -> for each swarm-secret-name -> ENV_VAR mapping, creates
     or rotates the Docker secret. Values are passed via ssh stdin only.
     Rotation uses the same sha256-checksum-label convention as
     deploy/create-secrets.sh (unchanged secrets are skipped silently).
 Safety properties:
  - FAILS HARD (non-zero) if any referenced env var is missing or empty, and
    lists the missing NAMES. A partial/broken render can never ship.
  - FAILS HARD if any unresolved placeholder remains after rendering.
  - NEVER prints a secret value — names and counts only.
  - Requires SWARM_MANAGER_IP in the environment and a usable ssh identity
    (both already set up by the provision-secrets step).
 Stacks not present in the manifest exit 0 with a notice, so this script is
 safe to call unconditionally; legacy case-entries in deploy.yml keep handling
 unmigrated stacks.
 """
 import hashlib
 import os
 import re
 import subprocess
 import sys
 REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 MANIFEST_PATH = os.path.join(REPO_ROOT, "secrets", "secrets-map.yaml")
 REMOTE_BASE = "/volume1/docker/compose-files"
 PLACEHOLDER_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
 SSH_OPTS = ["-o", "StrictHostKeyChecking=no"]
 def die(msg: str) -> None:
    print(f"ERROR: {msg}", file=sys.stderr)
    sys.exit(1)
 def load_manifest() -> dict:
    try:
        import yaml  # py3-yaml, installed by the provision-secrets step
    except ImportError:
        die("PyYAML not available — provision-secrets step must apk add py3-yaml")
    if not os.path.isfile(MANIFEST_PATH):
        die(f"manifest not found: {MANIFEST_PATH}")
    with open(MANIFEST_PATH, "r", encoding="utf-8") as fh:
        data = yaml.safe_load(fh) or {}
    stacks = data.get("stacks")
    if not isinstance(stacks, dict):
        die("manifest has no 'stacks:' mapping")
    return stacks
 def ssh_target() -> str:
    ip = os.environ.get("SWARM_MANAGER_IP", "").strip()
    if not ip:
        die("SWARM_MANAGER_IP is empty — check Woodpecker repo secrets")
    return f"root@{ip}"
 def ssh_run(target: str, remote_cmd: str, stdin_data: bytes | None = None,
            check: bool = True) -> subprocess.CompletedProcess:
    proc = subprocess.run(
        ["ssh", *SSH_OPTS, target, remote_cmd],
        input=stdin_data, capture_output=True,
    )
    if check and proc.returncode != 0:
        # stderr may be verbose but must never contain our secret values —
        # we only ever send values via stdin, never embed them in remote_cmd.
        die(f"remote command failed (rc={proc.returncode}): {remote_cmd}\n"
            f"{proc.stderr.decode(errors='replace').strip()}")
    return proc
 def render_template(template_path: str) -> str:
    if not os.path.isfile(template_path):
        die(f"env_template not found: {template_path}")
    with open(template_path, "r", encoding="utf-8") as fh:
        raw = fh.read()
    referenced = sorted(set(PLACEHOLDER_RE.findall(raw)))
    missing = [v for v in referenced
               if not os.environ.get(v, "").strip()]
    if missing:
        die("template references vars that are MISSING or EMPTY in the CI "
            "environment (add them via from_secret in "
            ".woodpecker/deploy.yml provision-secrets, and as Woodpecker "
            "secrets):\n  " + "\n  ".join(missing))
    rendered = PLACEHOLDER_RE.sub(lambda m: os.environ[m.group(1)], raw)
    # Belt-and-braces: nothing placeholder-shaped may survive the render.
    leftover = sorted(set(PLACEHOLDER_RE.findall(rendered)))
    if leftover:
        die("unresolved placeholders remain after rendering: "
            + ", ".join(leftover))
    print(f"  [render] {template_path}: {len(referenced)} secret placeholder(s) "
          f"resolved: {', '.join(referenced)}")
    return rendered
 def ship_env_file(target: str, rendered: str, dest_rel: str) -> None:
    dest = f"{REMOTE_BASE}/{dest_rel}"
    tmp = f"{dest}.provision-tmp"
    # Value travels over ssh stdin; never on a command line; atomic mv.
    ssh_run(target,
            f"umask 077 && cat > {tmp} && chmod 600 {tmp} && mv {tmp} {dest}",
            stdin_data=rendered.encode())
    keys = [ln.split("=", 1)[0] for ln in rendered.splitlines()
            if "=" in ln and not ln.lstrip().startswith("#") and ln.strip()]
    print(f"  [env] shipped {dest} ({len(keys)} keys): {', '.join(keys)}")
 def provision_docker_secret(target: str, name: str, env_var: str) -> None:
    value = os.environ.get(env_var, "")
    if not value.strip():
        die(f"docker secret '{name}': env var {env_var} is missing/empty")
    new_hash = hashlib.sha256(value.encode()).hexdigest()
    probe = ssh_run(
        target,
        f"docker secret inspect {name} "
        "--format '{{index .Spec.Labels \"checksum\"}}' 2>/dev/null || true",
        check=True)
    old_hash = probe.stdout.decode().strip()
    if old_hash == new_hash:
        print(f"  [skip]   docker secret {name} (unchanged)")
        return
    if old_hash:
        rm = ssh_run(target, f"docker secret rm {name}", check=False)
        if rm.returncode != 0:
            die(f"docker secret {name}: value changed but removal failed — "
                "it is probably referenced by a running service. Provision "
                "under a versioned name (see vaultwarden_database_url_v2 "
                "precedent) or scale the service down first.")
        action = "update"
    else:
        action = "create"
    ssh_run(target,
            f"docker secret create --label checksum={new_hash} "
            f"--label managed-by=woodpecker {name} -",
            stdin_data=value.encode())
    print(f"  [{action}] docker secret {name} (value via stdin)")
 def main() -> None:
    if len(sys.argv) != 2:
        die("usage: provision-stack.py <stack>")
    stack = sys.argv[1]
    stacks = load_manifest()
    cfg = stacks.get(stack)
    if cfg is None:
        print(f"  [info] stack '{stack}' not in secrets-map.yaml — "
              "legacy provisioning (deploy.yml case-entry) applies. Nothing to do.")
        return
    target = ssh_target()
    print(f"==> provision-stack: {stack}")
    template_rel = cfg.get("env_template")
    dest_rel = cfg.get("env_dest")
    if template_rel and not dest_rel:
        die("env_template set but env_dest missing in manifest")
    if template_rel:
        rendered = render_template(os.path.join(REPO_ROOT, template_rel))
        ship_env_file(target, rendered, dest_rel)
    for name, env_var in (cfg.get("docker_secrets") or {}).items():
        provision_docker_secret(target, name, env_var)
    print(f"==> provision-stack: {stack} done")
 if __name__ == "__main__":
    main()
@@ -20,14 +20,38 @@
 # All files in the folder matching *.yml or *.yaml are included.
 # Main file (<stack>.yml or <stack>.yaml) is always passed FIRST.
 # Remaining files are sorted and appended.
 #
 # Render pipeline (unified for all modes as of 2026-08-26):
 #   1. Render full compose YAML (with extras merged + env substituted) to a
 #      temp file.
 #   2. Run mount-guard.py against that temp file — checks every bind mount
 #      source path exists, and flags suspicious-looking empty Postgres data
 #      dirs, before anything touches Swarm.
 #   3. docker stack deploy -c <tempfile> <stack>
 #
 # Dollar-escaping (2026-08-26, see envparse.py comments for full detail):
 #   - Single-file path (no extras): envsubst does no '$' escaping of its
 #     own, and only Swarm's `docker stack deploy` interpolation pass runs
 #     downstream -> use export/export_merged (escapes '$' -> '$$' once).
 #   - Folder+extras path: `docker compose config` ALSO does its own '$'
 #     escaping on top of Swarm's -> use export_raw/export_raw_merged (no
 #     pre-escaping) or values get doubled twice. Getting this wrong
 #     silently corrupts any secret/hash containing '$' (confirmed impact:
 #     LITELLM keys truncated, IMMICH_KIOSK_BASICAUTH bcrypt hash mismatched).
 set -euo pipefail
 STACK="${1:?Usage: stack-deploy.sh <stack-name>}"
 DIR="/volume1/docker/compose-files"
 PY="$DIR/deploy/envparse.py"
 MOUNT_GUARD="$DIR/deploy/mount-guard.py"
 GLOBAL_ENV="$DIR/deploy/global.env"
 # ── Pre-flight: ensure local checkout is in sync with Gitea ─────────────────
 # Prevents deploying from a stale/diverged local tree (see incident 2026-08-26).
 # Invoked via `bash` explicitly so the tracked file's exec bit doesn't matter.
 bash "$DIR/deploy/git-guard.sh" || { echo "ERROR: git-guard check failed. Deploy aborted."; exit 1; }
 # ── Locate compose file(s) ──────────────────────────────────────────────────
 FOLDER="$DIR/$STACK"
@@ -80,6 +104,12 @@ for f in "${EXTRAS[@]:-}"; do
  [ -n "$f" ] && F_FLAGS+=(-f "$f")
 done
 # Whether the render will go through `docker compose config` (folder mode
 # with 1+ extras). This determines which escaping mode is correct — see
 # header comment and envparse.py for why these must differ.
 USES_COMPOSE_CONFIG=0
 [ "${#F_FLAGS[@]}" -gt 2 ] && USES_COMPOSE_CONFIG=1
 # ── Load env (global base + optional stack override) ─────────────────────────
 HINT="  Hint: 'secret not found' means Woodpecker hasn't provisioned secrets yet.\n  Trigger the pipeline: https://woodpecker.bryanmail.net\n"
@@ -89,19 +119,25 @@ STACK_EXISTS=0
 [ -f "$GLOBAL_ENV" ] && GLOBAL_EXISTS=1
 [ -f "$ENVFILE"    ] && STACK_EXISTS=1
 if [ "$USES_COMPOSE_CONFIG" -eq 1 ]; then
  EXPORT_MODE="export_raw"; EXPORT_MERGED_MODE="export_raw_merged"
 else
  EXPORT_MODE="export"; EXPORT_MERGED_MODE="export_merged"
 fi
 if [ "$GLOBAL_EXISTS" -eq 1 ] && [ "$STACK_EXISTS" -eq 1 ]; then
-  echo "    Env:     $GLOBAL_ENV + $ENVFILE (stack overrides global)"
+  echo "    Env:     $GLOBAL_ENV + $ENVFILE (stack overrides global) [$EXPORT_MERGED_MODE]"
-  eval "$(python3 "$PY" export_merged "$GLOBAL_ENV" "$ENVFILE")"
+  eval "$(python3 "$PY" "$EXPORT_MERGED_MODE" "$GLOBAL_ENV" "$ENVFILE")"
  VARS="$(python3 "$PY" vars_merged "$GLOBAL_ENV" "$ENVFILE")"
 elif [ "$GLOBAL_EXISTS" -eq 1 ]; then
-  echo "    Env:     $GLOBAL_ENV (no stack env)"
+  echo "    Env:     $GLOBAL_ENV (no stack env) [$EXPORT_MODE]"
-  eval "$(python3 "$PY" export "$GLOBAL_ENV")"
+  eval "$(python3 "$PY" "$EXPORT_MODE" "$GLOBAL_ENV")"
  VARS="$(python3 "$PY" vars "$GLOBAL_ENV")"
 elif [ "$STACK_EXISTS" -eq 1 ]; then
-  echo "    Env:     $ENVFILE (no global env)"
+  echo "    Env:     $ENVFILE (no global env) [$EXPORT_MODE]"
-  eval "$(python3 "$PY" export "$ENVFILE")"
+  eval "$(python3 "$PY" "$EXPORT_MODE" "$ENVFILE")"
  VARS="$(python3 "$PY" vars "$ENVFILE")"
 else
@@ -111,33 +147,44 @@ fi
 echo "    Vars:    $VARS"
-# ── Deploy ───────────────────────────────────────────────────────────────────
+# ── Render final compose YAML to a temp file ──────────────────────────────────
 RENDERED="$(mktemp /tmp/stack-deploy.XXXXXX.yml)"
 trap 'rm -f "$RENDERED"' EXIT
 if [ -n "$VARS" ]; then
-  if [ "${#F_FLAGS[@]}" -gt 2 ]; then
+  if [ "$USES_COMPOSE_CONFIG" -eq 1 ]; then
    # Folder mode with extras: merge via docker compose config
    docker compose "${F_FLAGS[@]}" config \
      | python3 "$PY" strip \
      | envsubst "$VARS" \
-      | docker stack deploy -c - "$STACK" \
+      > "$RENDERED"
      || { printf "\n%b" "$HINT"; exit 1; }
  else
    # Single file
    envsubst "$VARS" < "$MAIN" \
      | python3 "$PY" strip \
-      | docker stack deploy -c - "$STACK" \
+      > "$RENDERED"
      || { printf "\n%b" "$HINT"; exit 1; }
  fi
 else
-  if [ "${#F_FLAGS[@]}" -gt 2 ]; then
+  if [ "$USES_COMPOSE_CONFIG" -eq 1 ]; then
    docker compose "${F_FLAGS[@]}" config \
      | python3 "$PY" strip \
-      | docker stack deploy -c - "$STACK" \
+      > "$RENDERED"
      || { printf "\n%b" "$HINT"; exit 1; }
  else
-    docker stack deploy -c "$MAIN" "$STACK" \
+    python3 "$PY" strip < "$MAIN" > "$RENDERED"
      || { printf "\n%b" "$HINT"; exit 1; }
  fi
 fi
 # ── Pre-flight: bind mount paths exist + Postgres-data sanity ───────────────
 # See deploy/mount-guard.py for details. Blocks on missing paths or
 # suspicious-looking empty/uninitialized Postgres data directories (see
 # incident 2026-08-26: a wrong-but-existing empty bind path would have let
 # Postgres silently init a fresh DB while real data sat orphaned elsewhere).
 python3 "$MOUNT_GUARD" "$RENDERED" || { echo "ERROR: mount-guard check failed. Deploy aborted."; exit 1; }
 # ── Deploy ───────────────────────────────────────────────────────────────────
 docker stack deploy -c "$RENDERED" "$STACK" \
  || { printf "\n%b" "$HINT"; exit 1; }
 echo "==> Done: $STACK"
@@ -1,11 +1,25 @@
 # Immich stack environment variables - EXAMPLE
-# Copy to immich.env and fill in actual values
+# Copy this to immich.env (untracked, see .gitignore) and fill in
-# Pattern C (partial): DB_PASSWORD as Docker secret (_FILE), IMMICH_KIOSK_BASICAUTH in host .env (Traefik label)
+# IMMICH_KIOSK_BASICAUTH with a real value. Do NOT commit immich.env.
 #
 # Pattern C (partial): DB_PASSWORD as Docker secret (_FILE), IMMICH_KIOSK_BASICAUTH
 # in host .env (Traefik label).
 #
 # NOTE 2026-08-26: the paths below are the REAL, confirmed-correct values for
 # this homelab (verified via `docker service inspect immich_<svc> --format
 # '{{json .PreviousSpec.TaskTemplate.ContainerSpec.Mounts}}'` against actual
 # on-disk data). An earlier version of this file had plausible-looking but
 # WRONG generic paths (/volume1/docker/immich/db, /volume1/docker/immich/uploads),
 # which caused a real incident when immich.env was regenerated from this
 # template without checking against the live services first. If you ever
 # need to regenerate immich.env from this file, these paths should still be
 # correct — but if in doubt, re-verify with the PreviousSpec command above
 # before deploying.
 IMMICH_VERSION=release
-UPLOAD_LOCATION=/volume1/docker/immich/uploads
+UPLOAD_LOCATION=/volume1/Immich-Photos/
-BULK_UPLOAD_LOCATION=/volume1/docker/immich/bulk-uploads
+BULK_UPLOAD_LOCATION=/volume1/Immich-Photos/bulk-upload
-DB_DATA_LOCATION=/volume1/docker/immich/db
+DB_DATA_LOCATION=/volume1/docker/immich-postgresql
 REDIS_DATA_LOCATION=/volume1/docker/immich/redis
 DB_USERNAME=immich
@@ -20,4 +34,4 @@ REDIS_DBINDEX=0
 IMMICH_TRUSTED_PROXIES=172.16.0.0/12
 IMMICH_TRAEFIK_HOST=immich.bryanmail.net
 IMMICH_KIOSK_HOST=immich-kiosk.bryanmail.net
-IMMICH_KIOSK_BASICAUTH=user:password
+IMMICH_KIOSK_BASICAUTH=user:changeme
@@ -1,186 +0,0 @@
 {
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/mcpo_data/filesystem"
      ]
    },
    "memory": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-memory"
      ]
    },
    "proxmox-nuck7-1": {
      "command": "sh",
      "args": [
        "-c",
        "LOG_LEVEL=silent npx -y --package @bldg-7/proxmox-mcp@1.2.1 --package pino-pretty proxmox-mcp 2>/dev/null"
      ],
      "env": {
        "PROXMOX_HOST": "192.168.4.11",
        "PROXMOX_PORT": "8006",
        "PROXMOX_USER": "MCP@pve",
        "PROXMOX_TOKEN_NAME": "MCP",
        "PROXMOX_TOKEN_VALUE": "2052990e-749f-43f6-be7f-c9ad206281cc",
        "PROXMOX_SSL_MODE": "insecure",
        "PROXMOX_ALLOW_ELEVATED": "true",
        "PROXMOX_SSH_ENABLED": "true",
        "PROXMOX_SSH_HOST": "192.168.4.11",
        "PROXMOX_SSH_PORT": "22",
        "PROXMOX_SSH_USER": "root",
        "PROXMOX_SSH_KEY_PATH": "/app/ssh_keys/nuc-cluster",
        "PROXMOX_SSH_NODE": "nuck7-1",
        "PROXMOX_ALLOW_UNSAFE_COMMANDS": "true"
      }
    },
    "proxmox-nuck7-2": {
      "command": "sh",
      "args": [
        "-c",
        "LOG_LEVEL=silent npx -y --package @bldg-7/proxmox-mcp@1.2.1 --package pino-pretty proxmox-mcp 2>/dev/null"
      ],
      "env": {
        "PROXMOX_HOST": "192.168.4.12",
        "PROXMOX_PORT": "8006",
        "PROXMOX_USER": "MCP@pve",
        "PROXMOX_TOKEN_NAME": "MCP",
        "PROXMOX_TOKEN_VALUE": "2052990e-749f-43f6-be7f-c9ad206281cc",
        "PROXMOX_SSL_MODE": "insecure",
        "PROXMOX_ALLOW_ELEVATED": "true",
        "PROXMOX_SSH_ENABLED": "true",
        "PROXMOX_SSH_HOST": "192.168.4.12",
        "PROXMOX_SSH_PORT": "22",
        "PROXMOX_SSH_USER": "root",
        "PROXMOX_SSH_KEY_PATH": "/app/ssh_keys/nuc-cluster",
        "PROXMOX_SSH_NODE": "nuck7-2",
        "PROXMOX_ALLOW_UNSAFE_COMMANDS": "true"
      }
    },
    "proxmox-nuck7-3": {
      "command": "sh",
      "args": [
        "-c",
        "LOG_LEVEL=silent npx -y --package @bldg-7/proxmox-mcp@1.2.1 --package pino-pretty proxmox-mcp 2>/dev/null"
      ],
      "env": {
        "PROXMOX_HOST": "192.168.4.13",
        "PROXMOX_PORT": "8006",
        "PROXMOX_USER": "MCP@pve",
        "PROXMOX_TOKEN_NAME": "MCP",
        "PROXMOX_TOKEN_VALUE": "2052990e-749f-43f6-be7f-c9ad206281cc",
        "PROXMOX_SSL_MODE": "insecure",
        "PROXMOX_ALLOW_ELEVATED": "true",
        "PROXMOX_SSH_ENABLED": "true",
        "PROXMOX_SSH_HOST": "192.168.4.13",
        "PROXMOX_SSH_PORT": "22",
        "PROXMOX_SSH_USER": "root",
        "PROXMOX_SSH_KEY_PATH": "/app/ssh_keys/nuc-cluster",
        "PROXMOX_SSH_NODE": "nuck7-3",
        "PROXMOX_ALLOW_UNSAFE_COMMANDS": "true"
      }
    },
    "homeassistant": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://home.bryanmail.net/mcp_server/sse",
        "--header",
        "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIxNzhmYzI0NjA2Y2I0ZTg4ODI1N2VmMzdiZTNhY2E5YSIsImlhdCI6MTc3NDY4MzAxNiwiZXhwIjoyMDkwMDQzMDE2fQ.32kY2LVHzKZHWLc96T6z2P-8beNTnp2DHRUf2UEie2w"
      ]
    },
    "teams": {
      "command": "npx",
      "args": [
        "-y",
        "@floriscornel/teams-mcp@latest"
      ],
      "env": {
        "TEAMS_MCP_READ_ONLY": "true",
        "HOME": "/app/teams-mcp-auth"
      }
    },
    "ms365": {
      "command": "npx",
      "args": [
        "-y",
        "@softeria/ms-365-mcp-server@0.129.0",
        "--preset",
        "personal",
        "--read-only",
        "--discovery"
      ],
      "env": {
        "HOME": "/app/teams-mcp-auth",
        "MS365_MCP_CLIENT_ID": "14d82eec-204b-4c2f-b7e8-296a70dab67e",
        "MS365_MCP_TOKEN_CACHE_PATH": "/app/teams-mcp-auth/.teams-mcp-token-cache.json",
        "SILENT": "true"
      }
    },
    "unifi-network": {
      "command": "uvx",
      "args": [
        "unifi-network-mcp@latest"
      ],
      "env": {
        "UNIFI_HOST": "192.168.4.1",
        "UNIFI_USERNAME": "unifi-mcp",
        "UNIFI_PASSWORD": "3dHOEOMygTYeX3",
        "UNIFI_PORT": "443",
        "UNIFI_VERIFY_SSL": "false",
        "UV_CACHE_DIR": "/app/uv-cache"
      }
    },
    "authentik": {
      "command": "uvx",
      "args": [
        "authentik-diag-mcp",
        "--base-url",
        "https://auth.bryanmail.net",
        "--token",
        "LAm3lBTumOmsU8AiFQM2FmCZyoj8bTSR0FQnAcBy1QnTMEyU4oWozwdxTUap"
      ]
    },
    "gitea": {
      "command": "/mcpo_data/gitea-mcp",
      "args": [
        "-t",
        "stdio"
      ],
      "env": {
        "GITEA_HOST": "https://git.bryanmail.net",
        "GITEA_ACCESS_TOKEN": "5aa3a554c001b1dbe5215cb8cb388112801930e9"
      }
    },
    "gitea-admin": {
      "command": "/mcpo_data/gitea-mcp",
      "args": [
        "-t",
        "stdio"
      ],
      "env": {
        "GITEA_HOST": "https://git.bryanmail.net",
        "GITEA_ACCESS_TOKEN": "289225b0b8f1827242191874b2408db76af06321"
      }
    },
    "powerautomate": {
      "command": "npx",
      "args": [
        "-y",
        "powerautomate-mcp@latest",
        "--stdio"
      ],
      "env": {
        "HOME": "/app/powerautomate-auth",
        "PA_MCP_CLIENT_ID": "84b431ed-ef5d-48a0-b0a1-878cfdb71453",
        "PA_MCP_TENANT_ID": "0f6cf991-c449-480a-a71b-83003ce6edc1",
        "PA_CONFIG_PATH": "/app/powerautomate-auth/config.json"
      }
    }
  }
 }
@@ -1,217 +0,0 @@
 #!/usr/bin/env node
 // Minimal MCP server (stdio, JSON-RPC 2.0) for Woodpecker CI.
 // Zero dependencies — plain Node 18+ (built-in fetch). Config via env vars.
 //
 // Configuration (env):
 //   WOODPECKER_URL    e.g. https://your-woodpecker.example.com
 //   WOODPECKER_TOKEN  Woodpecker Personal Access Token (JWT)
 //
 // Tools:
 //   woodpecker_list_repos
 //   woodpecker_list_pipelines  { repo_id, limit? }
 //   woodpecker_get_pipeline    { repo_id, number }   -> status + workflow/step tree
 //   woodpecker_pipeline_logs   { repo_id, number, step_id, tail? } -> decoded step logs
 import { Buffer } from "node:buffer";
 const BASE = (process.env.WOODPECKER_URL || "").replace(/\/+$/, "");
 const TOKEN = process.env.WOODPECKER_TOKEN || "";
 function log(...a) {
  // Diagnostics go to stderr so they never corrupt the stdout JSON-RPC stream.
  process.stderr.write("[woodpecker-mcp] " + a.join(" ") + "\n");
 }
 async function api(path) {
  if (!BASE || !TOKEN) throw new Error("Missing WOODPECKER_URL or WOODPECKER_TOKEN in environment");
  const res = await fetch(`${BASE}/api${path}`, {
    headers: { Authorization: `Bearer ${TOKEN}`, Accept: "application/json" },
  });
  const body = await res.text();
  if (!res.ok) throw new Error(`HTTP ${res.status} ${path}: ${body.slice(0, 300)}`);
  return body ? JSON.parse(body) : null;
 }
 // --- tool response formatting ---
 function summarizePipeline(p) {
  const lines = [];
  lines.push(`pipeline #${p.number} (id=${p.id})  status=${p.status}  event=${p.event}  branch=${p.branch}`);
  lines.push(`commit=${(p.commit || "").slice(0, 12)}  author=${p.author}`);
  if (p.message) lines.push(`message: ${p.message.split("\n")[0]}`);
  for (const wf of p.workflows || []) {
    lines.push(`  WORKFLOW "${wf.name}" (pid=${wf.pid}) state=${wf.state}${wf.error ? ` error=${wf.error}` : ""}`);
    for (const c of wf.children || []) {
      lines.push(
        `    step "${c.name}" (id=${c.id}, pid=${c.pid}) ${c.state} exit=${c.exit_code ?? "-"} type=${c.type}`
      );
    }
  }
  return lines.join("\n");
 }
 function decodeLogs(entries) {
  if (!Array.isArray(entries)) return String(entries);
  return entries
    .map((e) => {
      const d = e?.data;
      if (d == null) return "";
      try {
        return Buffer.from(d, "base64").toString("utf-8");
      } catch {
        return String(d);
      }
    })
    .join("");
 }
 const TOOLS = [
  {
    name: "woodpecker_list_repos",
    description: "List repositories the token can access (id, full name, default branch).",
    inputSchema: { type: "object", properties: {}, additionalProperties: false },
    handler: async () => {
      const repos = await api(`/user/repos`);
      return (repos || [])
        .map((r) => `id=${r.id}  ${r.full_name}  default_branch=${r.default_branch}`)
        .join("\n") || "(no repositories)";
    },
  },
  {
    name: "woodpecker_list_pipelines",
    description: "Recent pipelines for a repository. Params: repo_id (number), limit (number, default 20).",
    inputSchema: {
      type: "object",
      properties: {
        repo_id: { type: "number", description: "Woodpecker repository ID" },
        limit: { type: "number", description: "How many pipelines to return (default 20)" },
      },
      required: ["repo_id"],
      additionalProperties: false,
    },
    handler: async ({ repo_id, limit }) => {
      const list = await api(`/repos/${repo_id}/pipelines?perPage=${limit || 20}`);
      return (list || [])
        .map(
          (p) =>
            `#${p.number}  ${p.status.padEnd(8)}  ${p.event.padEnd(12)}  ${p.branch}  ${(p.commit || "").slice(0, 8)}  ${(p.message || "").split("\n")[0]}`
        )
        .join("\n") || "(no pipelines)";
    },
  },
  {
    name: "woodpecker_get_pipeline",
    description: "Pipeline details: status plus the workflow/step tree (with step ids for fetching logs). Params: repo_id, number.",
    inputSchema: {
      type: "object",
      properties: {
        repo_id: { type: "number" },
        number: { type: "number", description: "Pipeline number (as shown in the UI)" },
      },
      required: ["repo_id", "number"],
      additionalProperties: false,
    },
    handler: async ({ repo_id, number }) => {
      const p = await api(`/repos/${repo_id}/pipelines/${number}`);
      return summarizePipeline(p);
    },
  },
  {
    name: "woodpecker_pipeline_logs",
    description: "Decoded logs for a single step. Params: repo_id, number (pipeline), step_id (from woodpecker_get_pipeline). Optional tail (last N lines).",
    inputSchema: {
      type: "object",
      properties: {
        repo_id: { type: "number" },
        number: { type: "number" },
        step_id: { type: "number", description: "Step id from woodpecker_get_pipeline" },
        tail: { type: "number", description: "Return only the last N lines (optional)" },
      },
      required: ["repo_id", "number", "step_id"],
      additionalProperties: false,
    },
    handler: async ({ repo_id, number, step_id, tail }) => {
      const entries = await api(`/repos/${repo_id}/logs/${number}/${step_id}`);
      let txt = decodeLogs(entries);
      if (tail && tail > 0) {
        txt = txt.split("\n").slice(-tail).join("\n");
      }
      return txt || "(no logs)";
    },
  },
 ];
 // --- JSON-RPC over stdio loop ---
 function send(msg) {
  process.stdout.write(JSON.stringify(msg) + "\n");
 }
 function reply(id, result) {
  send({ jsonrpc: "2.0", id, result });
 }
 function replyError(id, code, message) {
  send({ jsonrpc: "2.0", id, error: { code, message } });
 }
 async function handle(req) {
  const { id, method, params } = req;
  if (method === "initialize") {
    reply(id, {
      protocolVersion: params?.protocolVersion || "2024-11-05",
      capabilities: { tools: {} },
      serverInfo: { name: "woodpecker-mcp", version: "1.0.0" },
    });
    return;
  }
  if (method === "notifications/initialized" || method === "notifications/cancelled") {
    return; // notifications carry no response
  }
  if (method === "ping") {
    reply(id, {});
    return;
  }
  if (method === "tools/list") {
    reply(id, {
      tools: TOOLS.map((t) => ({ name: t.name, description: t.description, inputSchema: t.inputSchema })),
    });
    return;
  }
  if (method === "tools/call") {
    const tool = TOOLS.find((t) => t.name === params?.name);
    if (!tool) {
      replyError(id, -32602, `Unknown tool: ${params?.name}`);
      return;
    }
    try {
      const text = await tool.handler(params.arguments || {});
      reply(id, { content: [{ type: "text", text }] });
    } catch (e) {
      reply(id, { content: [{ type: "text", text: `Error: ${e.message}` }], isError: true });
    }
    return;
  }
  if (id !== undefined) replyError(id, -32601, `Unsupported method: ${method}`);
 }
 let buf = "";
 process.stdin.setEncoding("utf-8");
 process.stdin.on("data", (chunk) => {
  buf += chunk;
  let nl;
  while ((nl = buf.indexOf("\n")) >= 0) {
    const line = buf.slice(0, nl).trim();
    buf = buf.slice(nl + 1);
    if (!line) continue;
    let req;
    try {
      req = JSON.parse(line);
    } catch {
      continue;
    }
    handle(req).catch((e) => log("handler error:", e.message));
  }
 });
 process.stdin.on("end", () => process.exit(0));
 log("ready", BASE ? `(${BASE})` : "(WOODPECKER_URL is not set!)");
@@ -0,0 +1,38 @@
 # ─────────────────────────────────────────────────────────────────────────────
 # secrets-map.yaml — DATA-ONLY manifest for deploy/provision-stack.py
 #
 # RULES:
 #   - This file contains NO code, NO shell, NO secret values — only names.
 #   - Each stack entry declares:
 #       env_template:   repo path of the FULL env-file template (tracked).
 #                       The template is authoritative: the COMPLETE env file is
 #                       rendered from it on every provisioning run. Nothing is
 #                       line-edited in place, so keys can never silently go
 #                       missing.
 #       env_dest:       host path (relative to /volume1/docker/compose-files/)
 #                       the rendered env file is shipped to. Rendered file
 #                       exists ONLY on the host — never committed to git.
 #       docker_secrets: map of docker-swarm-secret-name -> CI ENV VAR NAME
 #                       (Pattern C). The env var must be declared via
 #                       from_secret: in .woodpecker/deploy.yml's
 #                       provision-secrets step (Woodpecker v3 requires explicit
 #                       per-secret declaration; there is no expose-all).
 #
 # ADDING A NEW SECRET (3 small steps, no shell edits):
 #   1. Add the secret value in Woodpecker UI (repo Settings -> Secrets).
 #   2. Declare it in .woodpecker/deploy.yml provision-secrets environment:
 #      block (from_secret) — mechanical two-line addition.
 #   3. Reference it here (docker_secrets:) and/or in the stack's
 #      .env.template as a dollar-brace placeholder.
 #
 # Stacks not listed here fall through to deploy.yml's legacy case-entries
 # untouched. Migration is deliberately one stack per PR.
 # ─────────────────────────────────────────────────────────────────────────────
 stacks:
  ai:
    env_template: ai/ai.env.template
    env_dest: ai/ai.env
    docker_secrets:
      flowagent_azure_client_id: FLOWAGENT_AZURE_CLIENT_ID
      flowagent_azure_tenant_id: FLOWAGENT_AZURE_TENANT_ID
      flowagent_azure_client_secret: FLOWAGENT_AZURE_CLIENT_SECRET
@@ -5,8 +5,8 @@ services:
      KEEPALIVED_INTERFACE: eth0
      KEEPALIVED_PRIORITY: "150"
      KEEPALIVED_ROUTER_ID: "51"
-      KEEPALIVED_PASSWORD: ${KEEPALIVED_PASSWORD}
+      KEEPALIVED_PASSWORD: "${KEEPALIVED_PASSWORD}"
-      KEEPALIVED_VIRTUAL_IPS: ${KEEPALIVED_VIRTUAL_IPS}
+      KEEPALIVED_VIRTUAL_IPS: "${KEEPALIVED_VIRTUAL_IPS}"
      KEEPALIVED_UNICAST_PEERS: "#PYTHON2BASH:['192.168.4.31', '192.168.4.32']"
    command: --copy-service
    cap_add: [NET_ADMIN, NET_BROADCAST, NET_RAW]
@@ -39,8 +39,8 @@ services:
      KEEPALIVED_INTERFACE: eth0
      KEEPALIVED_PRIORITY: "100"
      KEEPALIVED_ROUTER_ID: "51"
-      KEEPALIVED_PASSWORD: ${KEEPALIVED_PASSWORD}
+      KEEPALIVED_PASSWORD: "${KEEPALIVED_PASSWORD}"
-      KEEPALIVED_VIRTUAL_IPS: ${KEEPALIVED_VIRTUAL_IPS}
+      KEEPALIVED_VIRTUAL_IPS: "${KEEPALIVED_VIRTUAL_IPS}"
      KEEPALIVED_UNICAST_PEERS: "#PYTHON2BASH:['192.168.4.32', '192.168.4.33']"
    cap_add: [NET_ADMIN, NET_BROADCAST, NET_RAW]
    networks: [host]
@@ -147,7 +147,7 @@ services:
        order: stop-first
  traefik-certs-dumper:
-    image: ghcr.io/ldez/traefik-certs-dumper:v2.10.0
+    image: ghcr.io/ldez/traefik-certs-dumper:v2.11.4
    entrypoint: sh -c 'apk add jq ; while ! [ -e /letsencrypt/acme.json ] || ! [ `jq ".[] | .Certificates | length" /letsencrypt/acme.json` != 0 ]; do sleep 1 ; done && traefik-certs-dumper file --version v3 --watch --source /letsencrypt/acme.json --dest /letsencrypt/certs'
    volumes:
      - /volume1/docker/letsencrypt:/letsencrypt
Author	SHA1	Message	Date
Bot	eee6e543b1	Fix phantom verify failure: exclude secrets/ from stack detection + errexit-proof verify's stack-ps Root cause of PR #16's pipeline failure (#398-#401): secrets/ is a tooling folder (secrets-map.yaml, *.secrets.example), not a stack, but folder-detection (cut -d/ -f1, excluding only deploy/) computed ALL_STACKS="ai secrets" the first time a commit touched secrets/. The deploy step survived only because 'secrets' coincidentally sits in the bootstrap-tier skip list; verify had no guard — `docker stack ps secrets` exited non-zero and, under errexit, the OUTPUT=$(...) assignment killed the step before the designed WARNING path could run. The ai deploy itself was fully successful (complete 21-key render, all 6 services healthy) — the failure was pure phantom. Hazard first flagged July 2026 in PR #3 (closed unmerged). Changes: - grep -v '^deploy$' -> grep -vE '^(deploy\|secrets)$' in ALL five folder-detection sites (validate, provision-secrets x2, deploy, verify) - verify: `\|\| true` inside the stack-ps command substitution so a genuinely missing stack warns instead of aborting (comment added) - deploy step: also rsync secrets/ tooling to the host mirror unconditionally, same treatment as deploy/ - header: 2026-09-08 incident note Everything else byte-identical to main. Dollar-escape audit done.	2026-09-07 22:26:43 -07:00
AVB	be587de5be	Merge pull request 'Provisioning v2 (take 2): data-only manifest + full-file env template + generic provisioner — replaces #15 ' (#16 ) from secrets-provisioning-v3 into main ci/woodpecker/push/deploy Pipeline failed Details Reviewed-on: #16 https://ai.bryanmail.net/c/9f4c84c7-7aa1-4b24-a27e-98be6a5e737e	2026-09-07 22:06:03 -07:00
Bot	ef7e3d015e	provision-secrets: ai) case -> provision-stack.py + normalized env block (branch rebuilt on current main) Same two scoped changes as PR #15's rebase, now applied on a branch cut from TODAY's main so the merge base is current (PR #15 became unmergeable — its 09-03 merge base predated main's heavy hand-edits): 1. ai) case -> `python3 deploy/provision-stack.py ai` (one line). Retires main's heredoc with its live drift: AI_LITELLM_POSTGRES_PASSWORD printed twice (one copy from retired AI_LITELLM_DB_PASSWORD), stray legacy AI_WEBUI_SECRET_KEY, and the WEB_UI/WEBUI shell-var mismatch that renders an EMPTY OAuth client secret into ai.env. 2. env block ai section: normalized to exact template placeholder names, added AI_MCPO_API_KEY (NEW Woodpecker secret required pre-merge), pruned now-template-literal vars. apk gains python3 py3-yaml. Everything else byte-matches current main. Dollar-escape audit done per header lesson.	2026-09-07 21:47:04 -07:00
Bot	89a1e030c0	Add deploy/provision-stack.py — generic manifest-driven provisioner Carried over from secrets-provisioning-v2 (PR #15), content identical. Whole-file template render (hard-fail naming missing vars), env + Docker secrets shipped via ssh stdin only, sha256-checksum skip-if-unchanged, never prints a value. Stacks absent from the manifest exit 0 so legacy case-entries keep handling them.	2026-09-07 21:43:57 -07:00
Bot	e82b754386	Add ai/ai.env.template — full-file authoritative env template (AI_<SERVICE>_* naming) Carried over from secrets-provisioning-v2 (PR #15) post-rebase. Key names match exactly what current ai/ai.yaml references; MCPO_API_KEY stays unprefixed (ai.yaml consumes it unprefixed for both mcpo services). Placeholder names match the CI env vars declared in deploy.yml's provision-secrets block, including the normalized AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET (fixing main's WEB_UI/WEBUI mismatch that currently renders an EMPTY OAuth client secret into ai.env). NEW Woodpecker secret required before merge: ai_mcpo_api_key.	2026-09-07 21:43:05 -07:00
Bot	85734f4601	Add secrets/secrets-map.yaml — data-only provisioning manifest (ai stack first) Carried over from secrets-provisioning-v2 (PR #15), which became unmergeable after heavy parallel hand-edits to main moved the merge base too far. Content identical to v2.	2026-09-07 21:43:04 -07:00
AVB	264f4e7379	Change AWS_REGION_NAME variable ci/woodpecker/push/deploy Pipeline was successful Details ci/woodpecker/cron/renovate Pipeline was successful Details	2026-09-06 20:33:56 -07:00
AVB	9bbbd5fae3	Add AWS Region Name variable ci/woodpecker/push/deploy Pipeline was successful Details	2026-09-06 20:33:08 -07:00
AVB	e33fd8695f	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline was successful Details ci/woodpecker/cron/renovate Pipeline was successful Details	2026-09-06 10:59:05 -07:00
AVB	1828c89194	Update .woodpecker/deploy.yml ci/woodpecker/push/deploy Pipeline was successful Details	2026-09-06 10:53:55 -07:00
AVB	9ec5353d7e	Fixed LITELLM Database URL ci/woodpecker/push/deploy Pipeline was successful Details	2026-09-06 10:44:53 -07:00
AVB	dfa0d34a83	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline was successful Details	2026-09-06 10:29:27 -07:00
AVB	66638dd743	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline was successful Details	2026-09-06 01:29:02 -07:00
Bot	355ccada8c	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline failed Details ci/woodpecker/cron/renovate Pipeline was successful Details	2026-09-03 00:15:38 -07:00
Bot	096ebd6de5	Update .woodpecker/deploy.yml ci/woodpecker/push/deploy Pipeline was successful Details	2026-09-02 23:28:25 -07:00
Bot	980c54f84e	Update .woodpecker/deploy.yml	2026-09-02 23:27:19 -07:00
Bot	324a06ed2d	Update .woodpecker/deploy.yml	2026-09-02 23:26:30 -07:00
Bot	cee47822ec	Update .woodpecker/deploy.yml	2026-09-02 23:25:39 -07:00
Bot	68365746a6	Update .woodpecker/deploy.yml	2026-09-02 23:24:37 -07:00
Bot	eecfcf9367	Update .woodpecker/deploy.yml	2026-09-02 23:23:57 -07:00
Bot	d01c763354	Update .woodpecker/deploy.yml Redeploy after adding secret: secret "ai_open_web_ui_openid_provider_url" not found	2026-09-02 23:17:48 -07:00
Bot	0baee47b67	Fixed duplicate env: "AI_OPEN_WEB_UI_ENABLE_OAUTH_SIG" yaml: construct errors: line 190: mapping key "AI_OPEN_WEB_UI_ENABLE_OAUTH_SIGNUP" already defined at line 186	2026-09-02 23:02:51 -07:00
Bot	0435def8a0	Update .woodpecker/deploy.yml Added Open WebUI and LiteLLM secrets	2026-09-02 23:01:44 -07:00
Bot	ded255decc	Upgrade to open-webui:0.11.3 ci/woodpecker/push/deploy Pipeline was successful Details open-webui:0.11.1 > open-webui:0.11.3	2026-09-02 22:26:15 -07:00
Bot	ac091999b4	Merge pull request 'FlowAgent MCP cutover: point ai.yaml's mcpo at flowagent-mcpo image + wire Azure secrets (matched pair)' (#14 ) from flowagent-ai-cutover into main ci/woodpecker/push/deploy Pipeline failed Details Reviewed-on: #14 https://ai.bryanmail.net/c/9f4c84c7-7aa1-4b24-a27e-98be6a5e737e	2026-09-02 21:43:18 -07:00
Bot	a15e0100df	provision-secrets: add flowagent_* case-entry to the ai) stack (matched pair with ai.yaml's mcpo cutover) Surgical addition to the existing case "ai)" block only — every other case, comment, and line in this ~26KB pipeline file is byte-for-byte unchanged from main. Given this file's documented incident history from prior full-file rewrites (see header comment), this was hand-verified against the current main content line-by-line before writing, not generated fresh. Adds: - 3 new environment: entries (FLOWAGENT_AZURE_CLIENT_ID/TENANT_ID/ CLIENT_SECRET) via from_secret, alongside the existing ai_* secrets - Inside the existing `ai)` case, after the ai.env rewrite block: three create_or_update_secret calls provisioning flowagent_azure_client_id, flowagent_azure_tenant_id, flowagent_azure_client_secret as native Docker Swarm secrets (Pattern C, _FILE convention) — NOT written into ai/ai.env, since mcpo's flowagent entry reads them via /run/secrets/flowagent_azure_* per flowagent/entrypoint.sh, not env vars - A short comment explaining why these three are Pattern C secrets inside an otherwise Pattern-B (host .env) stack case This is the matched-pair companion to the ai.yaml commit in this same PR, per the FlowAgent progress note's explicit sequencing requirement (both land together, reviewed, not auto-merged to main).	2026-09-02 16:53:10 -07:00
Bot	bcfb24b16d	ai.yaml: cut mcpo over to the confirmed-working flowagent-mcpo image + wire flowagent Docker secrets (never touches mcpo-critical) Image pinned to git.bryanmail.net/homelab/flowagent-mcpo:916164714429 — the exact sha-tagged image built and smoke-tested clean in mcp-config pipeline #22 (build-and-push-flowagent + smoke-test-flowagent-image both success). NOT :latest, for reproducible deploys. Adds a secrets: block to the mcpo service only, referencing the 3 flowagent_azure_* Docker Swarm secrets (Pattern C, _FILE convention, consumed by flowagent/entrypoint.sh via /run/secrets/flowagent_azure_*). Declares those 3 secrets as external at top level, mirroring the existing vaultwarden_admin_token_v2 pattern in vaultwarden.yaml. Companion secret-provisioning change lands in the same PR as a separate commit in .woodpecker/deploy.yml's provision-secrets step (case "ai)"). mcpo-critical (lines defining Proxmox access) is completely untouched — still stock ghcr.io/open-webui/mcpo:main, no secrets: block, unchanged. Per FlowAgent MCP progress note: Azure AD App Registration is reported created with real secrets already in Woodpecker (both mcp-config and compose-files repos) as of this PR. This is a reviewed PR against a feature branch, NOT auto-merged to main — deploy only happens after manual review/merge, per user's explicit request for a manual-revert safety net on this high-blast-radius shared stack (LiteLLM, Open WebUI, n8n also live here).	2026-09-02 16:51:31 -07:00
Bot	42d31e38e4	git stack: switch provision-secrets to Pattern B .env rewrite (test phase) + add git to bootstrap-tier guard ci/woodpecker/push/deploy Pipeline was successful Details ci/woodpecker/cron/renovate Pipeline was successful Details Context: git hosts the source of truth for every other stack's compose files, so in a disaster-recovery scenario it must be restorable from a flat git.yaml + git.env backup alone, with zero dependency on a running Swarm's Docker secret store (native Docker secrets can't be resolved until Swarm already exists, which is the circular dependency git.yaml would otherwise create). Path 2 (documented decision): Pattern B for git specifically, same mechanism the ai stack already uses. provision-secrets / git) case: - Previously created native Docker secrets (git_db_password, git_runner_token, git_mcp_access_token) via create-secrets.sh — leftover from an earlier, abandoned Pattern C attempt. - Now rewrites only the secret-bearing lines (GITEA__database__PASSWD, GITEA_RUNNER_REGISTRATION_TOKEN, GITEA_MCP_ACCESS_TOKEN) via grep -v + printf, mirroring the ai) case exactly. Also strips the legacy GITEA_ACCESS_TOKEN key name so the test file converges on the git.env.example-documented key. - TEST PHASE: target is git.env.pipelinetest, NOT git.env. The real git.env is never opened for writing by this step. First run seeds the test file from the real git.env (to carry over all non-secret lines), then only the 3 secret lines are refreshed on every push. - Real cutover (pointing git.yaml/stack-deploy at the generated file, then retiring git.env.pipelinetest) is a deliberate follow-up step after manually diffing the rendered output. deploy / bootstrap-tier guard: - Added git to the traefik\|woodpecker\|postgresql\|secrets guard list. git.yaml changes now sync to the host but require a manual `stack-deploy.sh git` run, same as the other foundational stacks — prevents an auto-deploy of a bad git.yaml change from taking down Gitea before a human can look at it (Gitea itself is what every other pipeline needs to trigger a fix).	2026-09-02 00:02:36 -07:00
AVB	16a01a9ea5	Merge pull request 'Remove orphaned mcpo/ mirror (migrated to homelab/mcp-config)' (#13 ) from remove-orphaned-mcpo-mirror into main ci/woodpecker/push/deploy Pipeline failed Details ci/woodpecker/cron/renovate Pipeline was successful Details Reviewed-on: #13	2026-08-29 20:57:58 -07:00
admin	b6b23a7970	Remove mcpo/woodpecker-mcp.mjs — migrated unchanged to homelab/mcp-config	2026-08-29 20:46:01 -07:00
admin	150b262db3	Remove orphaned mcpo/config.json mirror — migrated to homelab/mcp-config This file was never actually deployed by this repo's pipeline (ai.yaml bind-mounts /volume1/docker/mcpo/config.json on the host directly; nothing here ever synced it there — see homelab/mcp-config README for full writeup). It also contained several real plaintext credentials (Proxmox token, Authentik JWT/token, UniFi password, Gitea tokens) that have since been rotated and templated in the new repo. NOTE: deleting this from HEAD does not remove it from compose-files' git history. The rotated credentials must be treated as permanently compromised regardless of this deletion.	2026-08-29 20:46:00 -07:00
admin	44babb6a22	Fix YAML alias-scanner crash: quote KEEPALIVED_PASSWORD/KEEPALIVED_VIRTUAL_IPS env values ci/woodpecker/push/deploy Pipeline was successful Details mount-guard.py failed with "expected alphabetic or numeric character, but found ''" during rendering. Root cause: KEEPALIVED_PASSWORD and KEEPALIVED_VIRTUAL_IPS were interpolated unquoted; when the live secret value happens to start with '', YAML's scanner parses it as an alias reference (*anchor) instead of literal text. KEEPALIVED_UNICAST_PEERS on the adjacent line was already quoted for the same reason (starts with '#') - this brings the other two env values in line with that existing convention. Also noted (not fixed here): live traefik-certs-dumper service is running v2.10.0 while this file already pins v2.11.4 - redeploy needed separately to pick that up.	2026-08-29 14:03:36 -07:00
AVB	10c35138db	Bump traefik-certs-dumper:v2.10.0 to 2.11.4 ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-29 13:56:35 -07:00
admin	162700bb4f	Fix: stack-deploy.sh now picks export_raw/export_raw_merged for folder+extras stacks (docker compose config path) vs export/export_merged for single-file stacks, matching envparse.py's dual-escaping fix ci/woodpecker/push/deploy Pipeline was successful Details ci/woodpecker/cron/renovate Pipeline was successful Details	2026-08-26 23:26:32 -07:00
admin	4af1209565	Fix: second double-interpolation bug in envparse.py - add export_raw/export_raw_merged (no escape_dollar) for the docker-compose-config render path, which already does its own dollar-escaping. Fixes IMMICH_KIOSK_BASICAUTH bcrypt hash corruption. ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 23:25:46 -07:00
admin	a557d9fb07	Refactor: unify stack-deploy.sh render paths into a single temp-file render step, then run mount-guard.py before docker stack deploy ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 23:04:11 -07:00
admin	f696efef11	Add: mount-guard.py - pre-deploy bind mount existence + Postgres empty-data heuristic check ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 23:03:11 -07:00
admin	2e889323ef	Fix: correct immich.env-example paths to match real on-disk data (UPLOAD_LOCATION, BULK_UPLOAD_LOCATION, DB_DATA_LOCATION) after 2026-08-26 incident where generic template paths were wrong ci/woodpecker/push/deploy Pipeline failed Details	2026-08-26 22:51:37 -07:00
admin	b620682a41	Fix: envparse.py strip mode now also drops top-level 'name:' key that docker compose config emits but Swarm's stack deploy schema rejects ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 22:37:38 -07:00
admin	87a254c902	Fix: envparse.py strip mode now collapses long-form depends_on mapping to Swarm-compatible short-form list ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 22:35:53 -07:00
admin	efd8caa218	Fix: invoke git-guard.sh via bash explicitly so tracked file mode bit doesn't matter ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 22:33:21 -07:00
admin	70814970c7	Harden .gitignore: broaden .env exclusion to .env (with explicit exceptions for global.env and .env-example templates), ignore stray .bak files ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 22:32:32 -07:00
admin	32e30f7e8b	Add: call git-guard.sh at top of stack-deploy.sh to enforce sync before deploy ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 22:31:12 -07:00
admin	0bc117c868	Add: git-guard.sh - pre-deploy sync check to prevent local/Gitea divergence ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 22:30:47 -07:00
AVB	7b5273b4b4	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 21:36:48 -07:00
AVB	9d0e9d9b60	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 21:28:03 -07:00
AVB	08949ba3a4	Update ai/ai.yaml ci/woodpecker/push/deploy Pipeline was successful Details	2026-08-26 21:24:17 -07:00
admin	ef4bcfcb25	ai: trigger redeploy to apply $$ escaping fix (PR #8 ) to litellm secrets ci/woodpecker/push/deploy Pipeline was successful Details Comment-only change. Forces a real deploy of the ai stack now that PR #8 (envparse.py $$ escaping) and PR #9 (deploy/ folder exclusion) are both merged, so litellm picks up the correctly-escaped LITELLM_MASTER_KEY/LITELLM_SALT_KEY instead of the truncated values currently running (truncated at the first literal '$' due to the envsubst+Compose double-interpolation bug fixed in #8).	2026-08-26 21:18:26 -07:00