admin
d209fc3222
Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation incident (2026-08-26): stack-deploy.sh's single-file deploy path is `envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`. envsubst embeds the raw secret value into the compose YAML text. If that value contains a literal '$' followed by word chars, the resulting YAML now contains what looks like a second variable reference. `docker stack deploy -c -` runs Compose's own interpolation pass on that text before creating the service, finds no such env var, and silently substitutes empty string -- truncating the secret in the running container with no error. Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm container as 73 chars, silently, on a real deploy. This is not specific to ai -- it affects every Pattern B stack (host .env + envsubst, not native Docker secrets): maintenance, media, unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of them could have a '$'-containing value truncating right now without detection, since the failure produces no warning. Fix: escape every literal '$' as '$$' in export/export_merged (which feed the `eval` that sets envsubst's actual source values), before envsubst ever sees them. envsubst does not interpret '$' in replacement text, so the doubled dollar survives envsubst intact; Compose's own interpolation pass then consumes exactly one level of escaping, landing on the correct single '$' with no leftover false variable reference. vars/vars_merged (envsubst's allowlist string, unrelated to values) are untouched. NOT deployed/merged yet -- pending review. The currently-running ai_litellm service still has the truncated keys and needs a fresh `stack-deploy.sh ai` run after this merges to pick up the corrected values.
86 lines
4.1 KiB
Python
86 lines
4.1 KiB
Python
import sys, re
|
|
|
|
def parse_env(path):
|
|
pairs = []
|
|
with open(path) as f:
|
|
for line in f:
|
|
line = line.strip()
|
|
if not line or line.startswith('#') or '=' not in line:
|
|
continue
|
|
k, _, v = line.partition('=')
|
|
k = k.strip(); v = v.strip()
|
|
if (v.startswith("'") and v.endswith("'")) or \
|
|
(v.startswith('"') and v.endswith('"')):
|
|
v = v[1:-1]
|
|
pairs.append((k, v))
|
|
return pairs
|
|
|
|
def merge_envs(base_path, override_path):
|
|
"""Merge two env files. Keys in override_path win over base_path."""
|
|
base = dict(parse_env(base_path))
|
|
override = dict(parse_env(override_path))
|
|
merged = {**base, **override}
|
|
return list(merged.items())
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# 2026-08-26 FIX — double-interpolation truncation bug (Pattern B stacks):
|
|
#
|
|
# stack-deploy.sh's single-file/no-extras path is:
|
|
# envsubst "$VARS" < ai.yaml | docker stack deploy -c - ai
|
|
#
|
|
# envsubst substitutes ${VAR} placeholders in the compose YAML with the
|
|
# literal, raw value of each shell-exported variable. If that raw value
|
|
# itself contains a literal '$' followed by word characters (e.g. a
|
|
# randomly-generated secret like "...i*Edu$RyAVYTqr4yzSS##..."), the
|
|
# resulting YAML text now contains what LOOKS like a second variable
|
|
# reference. `docker stack deploy -c -` runs Compose's own interpolation
|
|
# pass on that YAML text before creating the service — and Compose sees
|
|
# that leftover "$RyAVYTqr4yzSS", finds no such env var, and silently
|
|
# substitutes empty string. The secret gets truncated in the running
|
|
# container with NO error or warning.
|
|
#
|
|
# Confirmed impact (2026-08-26): LITELLM_MASTER_KEY and LITELLM_SALT_KEY
|
|
# in the `ai` stack were both truncated at their first literal '$' after
|
|
# a real deploy — 87-char secret arrived in the container as 73 chars.
|
|
#
|
|
# This affects every stack using Pattern B (host .env + envsubst, not
|
|
# native Docker secrets): ai, maintenance, media, unifi, guacamole,
|
|
# security, auth, traefik, meshcentral, ddm — any of them could have a
|
|
# '$'-containing value silently truncating right now without detection,
|
|
# since the failure is silent and only visible by diffing the source
|
|
# value against the live container env.
|
|
#
|
|
# Fix: escape every literal '$' in a value as '$$' at export time, BEFORE
|
|
# envsubst ever sees it. envsubst does not interpret '$' in the
|
|
# replacement text (only in the template), so the doubled dollar survives
|
|
# envsubst untouched. Compose's interpolation pass then consumes exactly
|
|
# one level of escaping ('$$' -> literal '$'), landing on the correct
|
|
# original single '$' with no leftover variable-reference lookalike.
|
|
#
|
|
# Only applied in export/export_merged (which feed `eval` to set the
|
|
# actual values envsubst reads) — NOT in vars/vars_merged, which just
|
|
# build envsubst's space-separated $VARNAME allowlist string and have
|
|
# nothing to do with actual values.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
def escape_dollar(v):
|
|
return v.replace('$', '$$')
|
|
|
|
mode = sys.argv[1]
|
|
if mode == 'export':
|
|
for k, v in parse_env(sys.argv[2]):
|
|
print('export {}={}'.format(k, repr(escape_dollar(v))))
|
|
elif mode == 'export_merged':
|
|
# export_merged <global.env> <stack.env>
|
|
for k, v in merge_envs(sys.argv[2], sys.argv[3]):
|
|
print('export {}={}'.format(k, repr(escape_dollar(v))))
|
|
elif mode == 'vars':
|
|
print(' '.join('$' + k for k, v in parse_env(sys.argv[2])))
|
|
elif mode == 'vars_merged':
|
|
# vars_merged <global.env> <stack.env>
|
|
print(' '.join('$' + k for k, v in merge_envs(sys.argv[2], sys.argv[3])))
|
|
elif mode == 'strip':
|
|
t = sys.stdin.read()
|
|
t = re.sub(r'[ \t]*env_file:[ \t]*\n([ \t]+-[^\n]*\n)+', '', t)
|
|
sys.stdout.write(t)
|