From d209fc32228bb203b5a002d97478791f39fb5d17 Mon Sep 17 00:00:00 2001 From: admin Date: Wed, 26 Aug 2026 20:56:23 -0700 Subject: [PATCH] fix: escape literal $ in .env values before envsubst (Pattern B stacks) Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation incident (2026-08-26): stack-deploy.sh's single-file deploy path is `envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`. envsubst embeds the raw secret value into the compose YAML text. If that value contains a literal '$' followed by word chars, the resulting YAML now contains what looks like a second variable reference. `docker stack deploy -c -` runs Compose's own interpolation pass on that text before creating the service, finds no such env var, and silently substitutes empty string -- truncating the secret in the running container with no error. Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm container as 73 chars, silently, on a real deploy. This is not specific to ai -- it affects every Pattern B stack (host .env + envsubst, not native Docker secrets): maintenance, media, unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of them could have a '$'-containing value truncating right now without detection, since the failure produces no warning. Fix: escape every literal '$' as '$$' in export/export_merged (which feed the `eval` that sets envsubst's actual source values), before envsubst ever sees them. envsubst does not interpret '$' in replacement text, so the doubled dollar survives envsubst intact; Compose's own interpolation pass then consumes exactly one level of escaping, landing on the correct single '$' with no leftover false variable reference. vars/vars_merged (envsubst's allowlist string, unrelated to values) are untouched. NOT deployed/merged yet -- pending review. The currently-running ai_litellm service still has the truncated keys and needs a fresh `stack-deploy.sh ai` run after this merges to pick up the corrected values. --- deploy/envparse.py | 48 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/deploy/envparse.py b/deploy/envparse.py index e5739d6..c74ad48 100644 --- a/deploy/envparse.py +++ b/deploy/envparse.py @@ -22,14 +22,58 @@ def merge_envs(base_path, override_path): merged = {**base, **override} return list(merged.items()) +# ───────────────────────────────────────────────────────────────────────────── +# 2026-08-26 FIX — double-interpolation truncation bug (Pattern B stacks): +# +# stack-deploy.sh's single-file/no-extras path is: +# envsubst "$VARS" < ai.yaml | docker stack deploy -c - ai +# +# envsubst substitutes ${VAR} placeholders in the compose YAML with the +# literal, raw value of each shell-exported variable. If that raw value +# itself contains a literal '$' followed by word characters (e.g. a +# randomly-generated secret like "...i*Edu$RyAVYTqr4yzSS##..."), the +# resulting YAML text now contains what LOOKS like a second variable +# reference. `docker stack deploy -c -` runs Compose's own interpolation +# pass on that YAML text before creating the service — and Compose sees +# that leftover "$RyAVYTqr4yzSS", finds no such env var, and silently +# substitutes empty string. The secret gets truncated in the running +# container with NO error or warning. +# +# Confirmed impact (2026-08-26): LITELLM_MASTER_KEY and LITELLM_SALT_KEY +# in the `ai` stack were both truncated at their first literal '$' after +# a real deploy — 87-char secret arrived in the container as 73 chars. +# +# This affects every stack using Pattern B (host .env + envsubst, not +# native Docker secrets): ai, maintenance, media, unifi, guacamole, +# security, auth, traefik, meshcentral, ddm — any of them could have a +# '$'-containing value silently truncating right now without detection, +# since the failure is silent and only visible by diffing the source +# value against the live container env. +# +# Fix: escape every literal '$' in a value as '$$' at export time, BEFORE +# envsubst ever sees it. envsubst does not interpret '$' in the +# replacement text (only in the template), so the doubled dollar survives +# envsubst untouched. Compose's interpolation pass then consumes exactly +# one level of escaping ('$$' -> literal '$'), landing on the correct +# original single '$' with no leftover variable-reference lookalike. +# +# Only applied in export/export_merged (which feed `eval` to set the +# actual values envsubst reads) — NOT in vars/vars_merged, which just +# build envsubst's space-separated $VARNAME allowlist string and have +# nothing to do with actual values. +# ───────────────────────────────────────────────────────────────────────────── + +def escape_dollar(v): + return v.replace('$', '$$') + mode = sys.argv[1] if mode == 'export': for k, v in parse_env(sys.argv[2]): - print('export {}={}'.format(k, repr(v))) + print('export {}={}'.format(k, repr(escape_dollar(v)))) elif mode == 'export_merged': # export_merged for k, v in merge_envs(sys.argv[2], sys.argv[3]): - print('export {}={}'.format(k, repr(v))) + print('export {}={}'.format(k, repr(escape_dollar(v)))) elif mode == 'vars': print(' '.join('$' + k for k, v in parse_env(sys.argv[2]))) elif mode == 'vars_merged':