Author SHA1 Message Date
admin f6fc288aa6 fix: exclude deploy/ from changed-stack detection
grep -E '^[^/.][^/]*/' matches ANY non-dot top-level folder in the
changed-files list, including deploy/ -- the shared tooling folder
synced by every deploy, not a stack. A PR touching only
deploy/envparse.py caused stack-deploy.sh to be invoked with "deploy"
as a stack name, which correctly errored ("No main compose file... in
.../deploy") since deploy/ has no deploy.yaml.

No live service was affected (the error occurs before any redeploy
attempt), but it produced a confusing FAIL on an otherwise-correct
change (PR #8) and could mask a real failure in the noise.

Adds `| grep -v '^deploy$'` after the folder-name extraction in all 5
places this pattern appears (validate, provision-secrets x2, deploy,
verify). deploy/ is already unconditionally rsynced at the top of the
deploy step regardless of which stacks changed, so excluding it from
the stack list is safe -- it will still be synced, just never treated
as a deployable stack.
2026-08-26 21:10:59 -07:00
AVB 701d1289ea Merge pull request 'fix: prevent envsubst+Compose double-interpolation from truncating $ secrets' (#8) from fix-dollar-double-interpolation into main
ci/woodpecker/push/deploy Pipeline failed
Reviewed-on: #8
https://ai.bryanmail.net/c/17c61a7d-f6cb-41f7-bd68-c17880df3646
2026-08-26 21:05:37 -07:00
admin d209fc3222 fix: escape literal $ in .env values before envsubst (Pattern B stacks)
Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation
incident (2026-08-26): stack-deploy.sh's single-file deploy path is
`envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`.
envsubst embeds the raw secret value into the compose YAML text. If
that value contains a literal '$' followed by word chars, the
resulting YAML now contains what looks like a second variable
reference. `docker stack deploy -c -` runs Compose's own interpolation
pass on that text before creating the service, finds no such env var,
and silently substitutes empty string -- truncating the secret in the
running container with no error.

Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm
container as 73 chars, silently, on a real deploy.

This is not specific to ai -- it affects every Pattern B stack (host
.env + envsubst, not native Docker secrets): maintenance, media,
unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of
them could have a '$'-containing value truncating right now without
detection, since the failure produces no warning.

Fix: escape every literal '$' as '$$' in export/export_merged (which
feed the `eval` that sets envsubst's actual source values), before
envsubst ever sees them. envsubst does not interpret '$' in replacement
text, so the doubled dollar survives envsubst intact; Compose's own
interpolation pass then consumes exactly one level of escaping,
landing on the correct single '$' with no leftover false variable
reference. vars/vars_merged (envsubst's allowlist string, unrelated to
values) are untouched.

NOT deployed/merged yet -- pending review. The currently-running
ai_litellm service still has the truncated keys and needs a fresh
`stack-deploy.sh ai` run after this merges to pick up the corrected
values.
2026-08-26 20:56:23 -07:00
admin d7fe56f6fe ai: add doc comment noting secrets are now provisioned via Woodpecker
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
No functional change -- this comment-only edit exists to trigger a real
deploy of the ai stack so provision-secrets' ai) case, and the AI_
var-name fix from PR #7, get exercised end-to-end for the first time.

Documents that MCPO_API_KEY is intentionally still manual/unmigrated.
2026-08-25 23:48:47 -07:00
3 changed files with 70 additions and 8 deletions
+19 -6
View File
@@ -36,6 +36,18 @@ when:
# until this was restored. Lesson: grep for the literal string '$${' and
# diff the count against the previous version before ever committing a
# full-file rewrite of this pipeline.
#
# 2026-08-26 FIX: FOLDER_STACKS/FOLDERS detection (grep -E '^[^/.][^/]*/')
# matches ANY non-dot top-level folder in the changed-files list, including
# deploy/ — the shared tooling folder, not a stack. A PR touching only
# deploy/envparse.py caused stack-deploy.sh to be invoked with "deploy" as
# a stack name, which correctly errored ("No main compose file... in
# .../deploy") since deploy/ has no deploy.yaml. No live service was
# affected (the error occurs before any redeploy), but it produced a
# confusing pipeline failure on an otherwise-correct change. deploy/ is
# already unconditionally rsynced at the top of the deploy step regardless
# of which stacks changed, so it's safe to exclude it from the stack list
# everywhere folders are detected below.
# ─────────────────────────────────────────────────────────────────────────────
steps:
@@ -53,8 +65,9 @@ steps:
FLAT=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.ya?ml$' || true)
# Folder: any file under a subfolder (e.g. immich/immich.yml).
# Exclude dotfolders (.woodpecker, .git, .gitea, etc.)
FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | sort -u || true)
# Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and deploy/
# (shared tooling, not a stack — see note above).
FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
[ -z "$FLAT" ] && [ -z "$FOLDERS" ] && echo "No stacks changed" && exit 0
@@ -180,14 +193,14 @@ steps:
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | sort -u || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && echo "No stacks changed, skipping" && exit 0
- scp -o StrictHostKeyChecking=no deploy/create-secrets.sh root@$${SWARM_MANAGER_IP}:/tmp/cs.sh
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | sort -u || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
for STACK in $ALL_STACKS; do
@@ -318,7 +331,7 @@ steps:
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | sort -u || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && echo "No stacks changed" && exit 0
@@ -372,7 +385,7 @@ steps:
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | sort -u || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && exit 0
+5
View File
@@ -1,3 +1,8 @@
# NOTE: litellm's AWS/DB/master-key secrets, open-webui's secret key/DB URL,
# and the OAuth client secret are provisioned into ai/ai.env at deploy time
# from Woodpecker secrets (see PRs #4, #6, #7) -- not committed here.
# MCPO_API_KEY (used by mcpo and mcpo-critical) is NOT yet migrated; it
# remains manually managed in ai/ai.env by design (see PR #4 discussion).
services:
open-webui:
image: ghcr.io/open-webui/open-webui:0.11.1
+46 -2
View File
@@ -22,14 +22,58 @@ def merge_envs(base_path, override_path):
merged = {**base, **override}
return list(merged.items())
# ─────────────────────────────────────────────────────────────────────────────
# 2026-08-26 FIX — double-interpolation truncation bug (Pattern B stacks):
#
# stack-deploy.sh's single-file/no-extras path is:
# envsubst "$VARS" < ai.yaml | docker stack deploy -c - ai
#
# envsubst substitutes ${VAR} placeholders in the compose YAML with the
# literal, raw value of each shell-exported variable. If that raw value
# itself contains a literal '$' followed by word characters (e.g. a
# randomly-generated secret like "...i*Edu$RyAVYTqr4yzSS##..."), the
# resulting YAML text now contains what LOOKS like a second variable
# reference. `docker stack deploy -c -` runs Compose's own interpolation
# pass on that YAML text before creating the service — and Compose sees
# that leftover "$RyAVYTqr4yzSS", finds no such env var, and silently
# substitutes empty string. The secret gets truncated in the running
# container with NO error or warning.
#
# Confirmed impact (2026-08-26): LITELLM_MASTER_KEY and LITELLM_SALT_KEY
# in the `ai` stack were both truncated at their first literal '$' after
# a real deploy — 87-char secret arrived in the container as 73 chars.
#
# This affects every stack using Pattern B (host .env + envsubst, not
# native Docker secrets): ai, maintenance, media, unifi, guacamole,
# security, auth, traefik, meshcentral, ddm — any of them could have a
# '$'-containing value silently truncating right now without detection,
# since the failure is silent and only visible by diffing the source
# value against the live container env.
#
# Fix: escape every literal '$' in a value as '$$' at export time, BEFORE
# envsubst ever sees it. envsubst does not interpret '$' in the
# replacement text (only in the template), so the doubled dollar survives
# envsubst untouched. Compose's interpolation pass then consumes exactly
# one level of escaping ('$$' -> literal '$'), landing on the correct
# original single '$' with no leftover variable-reference lookalike.
#
# Only applied in export/export_merged (which feed `eval` to set the
# actual values envsubst reads) — NOT in vars/vars_merged, which just
# build envsubst's space-separated $VARNAME allowlist string and have
# nothing to do with actual values.
# ─────────────────────────────────────────────────────────────────────────────
def escape_dollar(v):
return v.replace('$', '$$')
mode = sys.argv[1]
if mode == 'export':
for k, v in parse_env(sys.argv[2]):
print('export {}={}'.format(k, repr(v)))
print('export {}={}'.format(k, repr(escape_dollar(v))))
elif mode == 'export_merged':
# export_merged <global.env> <stack.env>
for k, v in merge_envs(sys.argv[2], sys.argv[3]):
print('export {}={}'.format(k, repr(v)))
print('export {}={}'.format(k, repr(escape_dollar(v))))
elif mode == 'vars':
print(' '.join('$' + k for k, v in parse_env(sys.argv[2])))
elif mode == 'vars_merged':