envsubst embeds the raw value of each .env variable directly into the rendered compose YAML. If that value contains a literal $ followed by word characters (exactly what a random secret generator produces — e.g. ...i*Edu$RyAVYTqr4yzSS##...), the resulting YAML text now contains what looks like a second, different variable reference.
docker stack deploy -c - then runs Compose's own interpolation pass on that YAML before creating the service. Compose sees the leftover $RyAVYTqr4yzSS, finds no such env var on the host, and silently substitutes empty string — no error, no warning, just a truncated secret in the running container.
Confirmed live impact (as of tonight)
ai_litellm's running LITELLM_MASTER_KEY is 73 chars; the correct value in ai/ai.env is 87 chars.
Same for LITELLM_SALT_KEY.
This predates and is unrelated to the AI secrets migration PRs (#4/#6/#7) — it's a latent bug in stack-deploy.sh's Pattern-B path that would have triggered on any deploy of ai once these particular key values happened to contain a $.
This affects every Pattern-B stack, not just ai: maintenance, media, unifi, guacamole, security, auth, traefik, meshcentral, ddm all use host .env + envsubst and could have $-containing values silently truncating right now, undetected, since there's no error output.
Fix
In envparse.py, escape every literal $ as $$ in export/export_merged output — the modes that feed the eval which actually sets the shell variables envsubst reads. envsubst doesn't interpret $ in replacement text (only in the template), so the doubled $$ survives envsubst intact. Compose's interpolation pass then consumes exactly one level of that escaping ($$ → literal $), landing back on the correct original value with no leftover false variable reference.
vars/vars_merged (which just build envsubst's space-separated $VARNAME allowlist string, not actual values) are untouched — they were never part of the bug.
Scope
One file, deploy/envparse.py. No changes to stack-deploy.sh, ai.yaml, or any secrets.
⚠️ Required manual follow-up after merge — separate from this PR
The currently runningai_litellm service still has the truncated LITELLM_MASTER_KEY/LITELLM_SALT_KEY right now — merging this PR alone doesn't fix a service that's already running with bad values. After merge, someone needs to run bash /volume1/docker/compose-files/deploy/stack-deploy.sh ai (or push any change under ai/) to force a fresh deploy that picks up the corrected escaping. I am not doing this automatically — flagging for your explicit go-ahead given tonight's incident.
Suggested verification before trusting this broadly
Before relying on this fix across all Pattern-B stacks, worth spot-checking one or two other stacks' currently-running secrets against their .env files for the same kind of silent truncation (e.g. grep '\$' */\*.env to find any other values containing literal $).
## ⚠️ Root-cause fix for tonight's `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` truncation incident
**Not merged/deployed yet — pending your review.** No service has been touched by this PR.
## Confirmed root cause
`stack-deploy.sh`'s deploy path is:
```
envsubst "$VARS" < ai.yaml | docker stack deploy -c - ai
```
`envsubst` embeds the **raw** value of each `.env` variable directly into the rendered compose YAML. If that value contains a literal `$` followed by word characters (exactly what a random secret generator produces — e.g. `...i*Edu$RyAVYTqr4yzSS##...`), the resulting YAML text now contains what *looks* like a second, different variable reference.
`docker stack deploy -c -` then runs **Compose's own interpolation pass** on that YAML before creating the service. Compose sees the leftover `$RyAVYTqr4yzSS`, finds no such env var on the host, and **silently substitutes empty string** — no error, no warning, just a truncated secret in the running container.
## Confirmed live impact (as of tonight)
- `ai_litellm`'s running `LITELLM_MASTER_KEY` is 73 chars; the correct value in `ai/ai.env` is 87 chars.
- Same for `LITELLM_SALT_KEY`.
- This predates and is unrelated to the AI secrets migration PRs (#4/#6/#7) — it's a latent bug in `stack-deploy.sh`'s Pattern-B path that would have triggered on *any* deploy of `ai` once these particular key values happened to contain a `$`.
- **This affects every Pattern-B stack**, not just `ai`: `maintenance`, `media`, `unifi`, `guacamole`, `security`, `auth`, `traefik`, `meshcentral`, `ddm` all use host `.env` + `envsubst` and could have `$`-containing values silently truncating right now, undetected, since there's no error output.
## Fix
In `envparse.py`, escape every literal `$` as `$$` in `export`/`export_merged` output — the modes that feed the `eval` which actually sets the shell variables `envsubst` reads. `envsubst` doesn't interpret `$` in *replacement* text (only in the template), so the doubled `$$` survives `envsubst` intact. Compose's interpolation pass then consumes exactly one level of that escaping (`$$` → literal `$`), landing back on the correct original value with no leftover false variable reference.
`vars`/`vars_merged` (which just build `envsubst`'s space-separated `$VARNAME` allowlist string, not actual values) are untouched — they were never part of the bug.
## Scope
One file, `deploy/envparse.py`. No changes to `stack-deploy.sh`, `ai.yaml`, or any secrets.
## ⚠️ Required manual follow-up after merge — separate from this PR
The **currently running** `ai_litellm` service still has the truncated `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` right now — merging this PR alone doesn't fix a service that's already running with bad values. After merge, someone needs to run `bash /volume1/docker/compose-files/deploy/stack-deploy.sh ai` (or push any change under `ai/`) to force a fresh deploy that picks up the corrected escaping. **I am not doing this automatically** — flagging for your explicit go-ahead given tonight's incident.
## Suggested verification before trusting this broadly
Before relying on this fix across all Pattern-B stacks, worth spot-checking one or two other stacks' currently-running secrets against their `.env` files for the same kind of silent truncation (e.g. `grep '\$' */\*.env` to find any other values containing literal `$`).
Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation
incident (2026-08-26): stack-deploy.sh's single-file deploy path is
`envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`.
envsubst embeds the raw secret value into the compose YAML text. If
that value contains a literal '$' followed by word chars, the
resulting YAML now contains what looks like a second variable
reference. `docker stack deploy -c -` runs Compose's own interpolation
pass on that text before creating the service, finds no such env var,
and silently substitutes empty string -- truncating the secret in the
running container with no error.
Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm
container as 73 chars, silently, on a real deploy.
This is not specific to ai -- it affects every Pattern B stack (host
.env + envsubst, not native Docker secrets): maintenance, media,
unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of
them could have a '$'-containing value truncating right now without
detection, since the failure produces no warning.
Fix: escape every literal '$' as '$$' in export/export_merged (which
feed the `eval` that sets envsubst's actual source values), before
envsubst ever sees them. envsubst does not interpret '$' in replacement
text, so the doubled dollar survives envsubst intact; Compose's own
interpolation pass then consumes exactly one level of escaping,
landing on the correct single '$' with no leftover false variable
reference. vars/vars_merged (envsubst's allowlist string, unrelated to
values) are untouched.
NOT deployed/merged yet -- pending review. The currently-running
ai_litellm service still has the truncated keys and needs a fresh
`stack-deploy.sh ai` run after this merges to pick up the corrected
values.
AVB
merged commit 701d1289ea into main2026-08-26 21:05:37 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
⚠️ Root-cause fix for tonight's
LITELLM_MASTER_KEY/LITELLM_SALT_KEYtruncation incidentNot merged/deployed yet — pending your review. No service has been touched by this PR.
Confirmed root cause
stack-deploy.sh's deploy path is:envsubstembeds the raw value of each.envvariable directly into the rendered compose YAML. If that value contains a literal$followed by word characters (exactly what a random secret generator produces — e.g....i*Edu$RyAVYTqr4yzSS##...), the resulting YAML text now contains what looks like a second, different variable reference.docker stack deploy -c -then runs Compose's own interpolation pass on that YAML before creating the service. Compose sees the leftover$RyAVYTqr4yzSS, finds no such env var on the host, and silently substitutes empty string — no error, no warning, just a truncated secret in the running container.Confirmed live impact (as of tonight)
ai_litellm's runningLITELLM_MASTER_KEYis 73 chars; the correct value inai/ai.envis 87 chars.LITELLM_SALT_KEY.stack-deploy.sh's Pattern-B path that would have triggered on any deploy ofaionce these particular key values happened to contain a$.ai:maintenance,media,unifi,guacamole,security,auth,traefik,meshcentral,ddmall use host.env+envsubstand could have$-containing values silently truncating right now, undetected, since there's no error output.Fix
In
envparse.py, escape every literal$as$$inexport/export_mergedoutput — the modes that feed theevalwhich actually sets the shell variablesenvsubstreads.envsubstdoesn't interpret$in replacement text (only in the template), so the doubled$$survivesenvsubstintact. Compose's interpolation pass then consumes exactly one level of that escaping ($$→ literal$), landing back on the correct original value with no leftover false variable reference.vars/vars_merged(which just buildenvsubst's space-separated$VARNAMEallowlist string, not actual values) are untouched — they were never part of the bug.Scope
One file,
deploy/envparse.py. No changes tostack-deploy.sh,ai.yaml, or any secrets.⚠️ Required manual follow-up after merge — separate from this PR
The currently running
ai_litellmservice still has the truncatedLITELLM_MASTER_KEY/LITELLM_SALT_KEYright now — merging this PR alone doesn't fix a service that's already running with bad values. After merge, someone needs to runbash /volume1/docker/compose-files/deploy/stack-deploy.sh ai(or push any change underai/) to force a fresh deploy that picks up the corrected escaping. I am not doing this automatically — flagging for your explicit go-ahead given tonight's incident.Suggested verification before trusting this broadly
Before relying on this fix across all Pattern-B stacks, worth spot-checking one or two other stacks' currently-running secrets against their
.envfiles for the same kind of silent truncation (e.g.grep '\$' */\*.envto find any other values containing literal$).