fix: escape literal $ in .env values before envsubst (Pattern B stacks)

Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation
incident (2026-08-26): stack-deploy.sh's single-file deploy path is
`envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`.
envsubst embeds the raw secret value into the compose YAML text. If
that value contains a literal '$' followed by word chars, the
resulting YAML now contains what looks like a second variable
reference. `docker stack deploy -c -` runs Compose's own interpolation
pass on that text before creating the service, finds no such env var,
and silently substitutes empty string -- truncating the secret in the
running container with no error.

Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm
container as 73 chars, silently, on a real deploy.

This is not specific to ai -- it affects every Pattern B stack (host
.env + envsubst, not native Docker secrets): maintenance, media,
unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of
them could have a '$'-containing value truncating right now without
detection, since the failure produces no warning.

Fix: escape every literal '$' as '$$' in export/export_merged (which
feed the `eval` that sets envsubst's actual source values), before
envsubst ever sees them. envsubst does not interpret '$' in replacement
text, so the doubled dollar survives envsubst intact; Compose's own
interpolation pass then consumes exactly one level of escaping,
landing on the correct single '$' with no leftover false variable
reference. vars/vars_merged (envsubst's allowlist string, unrelated to
values) are untouched.

NOT deployed/merged yet -- pending review. The currently-running
ai_litellm service still has the truncated keys and needs a fresh
`stack-deploy.sh ai` run after this merges to pick up the corrected
values.

.woodpecker/deploy.yml

@@ -26,6 +26,16 @@ when:
 #   This was the root cause of a long-running "SWARM_MANAGER_IP secret is
 #   empty" failure: braced references were blanked at compile time before the
 #   shell ever saw them.
+#
+#   2026-08-26 HOTFIX: a full-file rewrite (AI secrets migration PR) dropped
+#   one $ from every $${VAR} occurrence throughout this file, re-introducing
+#   exactly the bug described above for EVERY secret-backed var, not just the
+#   new AI ones. The if-empty guards caught it immediately (SWARM_MANAGER_IP
+#   came back blank) and aborted before any ssh/scp/rsync ran, so no live
+#   secret or service was touched — but no CI provisioning/deploy could run
+#   until this was restored. Lesson: grep for the literal string '$${' and
+#   diff the count against the previous version before ever committing a
+#   full-file rewrite of this pipeline.
 # ─────────────────────────────────────────────────────────────────────────────
 
 steps:
@@ -140,6 +150,22 @@ steps:
         from_secret: entertainment_sparky_encryption_key
       ENTERTAINMENT_BETTER_AUTH_SECRET:
         from_secret: entertainment_better_auth_secret
+      AI_AWS_ACCESS_KEY_ID:
+        from_secret: ai_aws_access_key_id
+      AI_AWS_SECRET_ACCESS_KEY:
+        from_secret: ai_aws_secret_access_key
+      AI_LITELLM_MASTER_KEY:
+        from_secret: ai_litellm_master_key
+      AI_LITELLM_SALT_KEY:
+        from_secret: ai_litellm_salt_key
+      AI_LITELLM_DB_PASSWORD:
+        from_secret: ai_litellm_db_password
+      AI_WEBUI_SECRET_KEY:
+        from_secret: ai_webui_secret_key
+      AI_OPEN_WEBUI_DATABASE_URL:
+        from_secret: ai_open_webui_database_url
+      AI_OAUTH_CLIENT_SECRET:
+        from_secret: ai_oauth_client_secret
     commands:
       - apk add --no-cache openssh-client
       - mkdir -p ~/.ssh
@@ -222,7 +248,40 @@ steps:
                 create_or_update_secret 'vaultwarden_admin_token'     '$${VAULTWARDEN_ADMIN_TOKEN}'
                 create_or_update_secret 'vaultwarden_database_url_v2' '$${VAULTWARDEN_DATABASE_URL}'";;
             ai)
-              echo "  No Docker secrets for ai -- secrets in host .env";;
+              # NOTE: ai stack uses Pattern B (host .env, not native Docker secrets) —
+              # LiteLLM/boto3 and Open WebUI don't support the _FILE convention for these
+              # vars. Instead of Docker secrets, we regenerate ONLY the migrated lines in
+              # the remote ai/ai.env in place via grep -v + printf (never sed, since values
+              # may contain slash, dollar sign, ampersand). All other lines — including
+              # MCPO_API_KEY, OAUTH_CLIENT_ID, WEBUI_URL, and other non-secret config — are
+              # left completely untouched. MCPO_API_KEY migration is deferred to a
+              # follow-up; this step never reads or writes it.
+              #
+              # IMPORTANT: ai.yaml's litellm service references ${AI_AWS_ACCESS_KEY_ID} /
+              # ${AI_AWS_SECRET_ACCESS_KEY} (AI_-prefixed) and renders them into the
+              # container as plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (see commit
+              # 37ed671a, "Change AWS keys to use Woodpecker Secrets"). So ai.env must be
+              # written with the AI_-prefixed key names, NOT the plain ones — writing
+              # plain AWS_ACCESS_KEY_ID here would leave ${AI_AWS_ACCESS_KEY_ID}
+              # unresolved at compose-render time (empty), silently breaking Bedrock auth.
+              # All other migrated vars in ai.yaml use plain (unprefixed) names, so only
+              # these two lines need the AI_ prefix.
+              ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/ai/ai.env
+                TMP=\$FILE.tmp.\$\$
+                grep -vE '^(AI_AWS_ACCESS_KEY_ID|AI_AWS_SECRET_ACCESS_KEY|LITELLM_MASTER_KEY|LITELLM_SALT_KEY|POSTGRES_PASSWORD|DATABASE_URL|WEBUI_SECRET_KEY|OPEN_WEBUI_DATABASE_URL|OAUTH_CLIENT_SECRET)=' \$FILE > \$TMP 2>/dev/null || touch \$TMP
+                { cat \$TMP
+                  printf 'AI_AWS_ACCESS_KEY_ID=%s\n' '$${AI_AWS_ACCESS_KEY_ID}'
+                  printf 'AI_AWS_SECRET_ACCESS_KEY=%s\n' '$${AI_AWS_SECRET_ACCESS_KEY}'
+                  printf 'LITELLM_MASTER_KEY=%s\n' '$${AI_LITELLM_MASTER_KEY}'
+                  printf 'LITELLM_SALT_KEY=%s\n' '$${AI_LITELLM_SALT_KEY}'
+                  printf 'POSTGRES_PASSWORD=%s\n' '$${AI_LITELLM_DB_PASSWORD}'
+                  printf 'DATABASE_URL=postgresql://LiteLLM:%s@postgresql:5432/litellm\n' '$${AI_LITELLM_DB_PASSWORD}'
+                  printf 'WEBUI_SECRET_KEY=%s\n' '$${AI_WEBUI_SECRET_KEY}'
+                  printf 'OPEN_WEBUI_DATABASE_URL=%s\n' '$${AI_OPEN_WEBUI_DATABASE_URL}'
+                  printf 'OAUTH_CLIENT_SECRET=%s\n' '$${AI_OAUTH_CLIENT_SECRET}'
+                } > \$FILE
+                rm -f \$TMP
+                echo '  [OK] ai/ai.env secrets updated'";;
             entertainment)
               ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
                 create_or_update_secret 'entertainment_discord_token'          '$${ENTERTAINMENT_DISCORD_TOKEN}'
@@ -316,12 +375,36 @@ steps:
         FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | sort -u || true)
         ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
         [ -z "$ALL_STACKS" ] && exit 0
-        sleep 5
+
+        # NOTE: `docker stack deploy` briefly tears down and recreates tasks in
+        # Swarm's internal bookkeeping, so `docker stack ps` can transiently
+        # return nothing right after deploy even when the service is healthy.
+        # A single `sleep 5` + one-shot check produced false-alarm-looking
+        # "nothing found in stack" output on ordinary deploys (e.g. vaultwarden,
+        # 2026-08-25). Retry with backoff instead of a single fixed sleep, and
+        # only warn (don't fail the pipeline) if tasks never show up.
+        ATTEMPTS=6
+        DELAY=5
         for STACK in $ALL_STACKS; do
           echo "--- $STACK ---"
-          ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
+          i=1
+          while [ "$i" -le "$ATTEMPTS" ]; do
+            OUTPUT=$(ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
               "docker stack ps $STACK --filter desired-state=running \
-            --format '  {{.Name}} {{.CurrentState}}'"
+              --format '  {{.Name}} {{.CurrentState}}'" 2>/dev/null)
+            if [ -n "$OUTPUT" ]; then
+              echo "$OUTPUT"
+              break
+            fi
+            if [ "$i" -eq "$ATTEMPTS" ]; then
+              echo "  WARNING: no running tasks found for $STACK after $((ATTEMPTS * DELAY))s."
+              echo "  This may be transient Swarm settle time, or a real problem — check manually:"
+              echo "  ssh root@$${SWARM_MANAGER_IP} 'docker stack ps $STACK --no-trunc'"
+            else
+              sleep "$DELAY"
+            fi
+            i=$((i + 1))
+          done
         done
 
   notify-success:

SECRETS.md

@@ -14,3 +14,8 @@
 # 2. Select repository: homelab/compose-files
 # 3. Settings → Secrets
 # 4. Add SSH_KEY and TEAMS_WEBHOOK
+
+# ai stack secrets (added 2026-08-25, see PR #4):
+#   ai_aws_access_key_id, ai_aws_secret_access_key, ai_litellm_master_key,
+#   ai_litellm_salt_key, ai_litellm_db_password, ai_webui_secret_key,
+#   ai_open_webui_database_url, ai_oauth_client_secret

ai/ai.yaml

@@ -1,3 +1,8 @@
+# NOTE: litellm's AWS/DB/master-key secrets, open-webui's secret key/DB URL,
+# and the OAuth client secret are provisioned into ai/ai.env at deploy time
+# from Woodpecker secrets (see PRs #4, #6, #7) -- not committed here.
+# MCPO_API_KEY (used by mcpo and mcpo-critical) is NOT yet migrated; it
+# remains manually managed in ai/ai.env by design (see PR #4 discussion).
 services:
   open-webui:
     image: ghcr.io/open-webui/open-webui:0.11.1

deploy/envparse.py

@@ -22,14 +22,58 @@ def merge_envs(base_path, override_path):
     merged = {**base, **override}
     return list(merged.items())
 
+# ─────────────────────────────────────────────────────────────────────────────
+# 2026-08-26 FIX — double-interpolation truncation bug (Pattern B stacks):
+#
+#   stack-deploy.sh's single-file/no-extras path is:
+#     envsubst "$VARS" < ai.yaml | docker stack deploy -c - ai
+#
+#   envsubst substitutes ${VAR} placeholders in the compose YAML with the
+#   literal, raw value of each shell-exported variable. If that raw value
+#   itself contains a literal '$' followed by word characters (e.g. a
+#   randomly-generated secret like "...i*Edu$RyAVYTqr4yzSS##..."), the
+#   resulting YAML text now contains what LOOKS like a second variable
+#   reference. `docker stack deploy -c -` runs Compose's own interpolation
+#   pass on that YAML text before creating the service — and Compose sees
+#   that leftover "$RyAVYTqr4yzSS", finds no such env var, and silently
+#   substitutes empty string. The secret gets truncated in the running
+#   container with NO error or warning.
+#
+#   Confirmed impact (2026-08-26): LITELLM_MASTER_KEY and LITELLM_SALT_KEY
+#   in the `ai` stack were both truncated at their first literal '$' after
+#   a real deploy — 87-char secret arrived in the container as 73 chars.
+#
+#   This affects every stack using Pattern B (host .env + envsubst, not
+#   native Docker secrets): ai, maintenance, media, unifi, guacamole,
+#   security, auth, traefik, meshcentral, ddm — any of them could have a
+#   '$'-containing value silently truncating right now without detection,
+#   since the failure is silent and only visible by diffing the source
+#   value against the live container env.
+#
+#   Fix: escape every literal '$' in a value as '$$' at export time, BEFORE
+#   envsubst ever sees it. envsubst does not interpret '$' in the
+#   replacement text (only in the template), so the doubled dollar survives
+#   envsubst untouched. Compose's interpolation pass then consumes exactly
+#   one level of escaping ('$$' -> literal '$'), landing on the correct
+#   original single '$' with no leftover variable-reference lookalike.
+#
+#   Only applied in export/export_merged (which feed `eval` to set the
+#   actual values envsubst reads) — NOT in vars/vars_merged, which just
+#   build envsubst's space-separated $VARNAME allowlist string and have
+#   nothing to do with actual values.
+# ─────────────────────────────────────────────────────────────────────────────
+
+def escape_dollar(v):
+    return v.replace('$', '$$')
+
 mode = sys.argv[1]
 if mode == 'export':
     for k, v in parse_env(sys.argv[2]):
-        print('export {}={}'.format(k, repr(v)))
+        print('export {}={}'.format(k, repr(escape_dollar(v))))
 elif mode == 'export_merged':
     # export_merged <global.env> <stack.env>
     for k, v in merge_envs(sys.argv[2], sys.argv[3]):
-        print('export {}={}'.format(k, repr(v)))
+        print('export {}={}'.format(k, repr(escape_dollar(v))))
 elif mode == 'vars':
     print(' '.join('$' + k for k, v in parse_env(sys.argv[2])))
 elif mode == 'vars_merged':