ai/ai.yaml's litellm service (as of commit 37ed671a, "Change AWS keys to use Woodpecker Secrets", pushed directly to main ahead of the secrets-migration PR) references:
i.e. it expects the AI_-prefixed host var names, renamed to plain AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY inside the container.
The ai) provisioning case in deploy.yml (added in the earlier AI secrets migration) wrote the plain, unprefixed names into ai/ai.env instead. Every other migrated var in ai.yaml (LITELLM_MASTER_KEY, DATABASE_URL, WEBUI_SECRET_KEY, etc.) uses plain names — only these two AWS vars use the AI_ prefix — so this mismatch wasn't caught by pattern-matching against the rest of the file.
Impact if unfixed
On the next deploy of the ai stack, docker compose config would render ${AI_AWS_ACCESS_KEY_ID} as an empty string (since only the plain-named var would exist in the exported env), silently breaking Bedrock/AWS auth in litellm — no error, just empty credentials passed to boto3.
Fix
Changed both the grep -vE exclusion pattern and the two printf lines in the ai) case to use AI_AWS_ACCESS_KEY_ID / AI_AWS_SECRET_ACCESS_KEY instead of the plain names. Added an inline comment explaining why these two vars specifically need the prefix while the rest don't, to prevent this from silently regressing again in a future rewrite.
Scope
Only the ai) case's two AWS printf lines + the exclusion regex changed. No other stacks, no other AI vars affected.
Verification after merge
This is the fix needed before the pending "trigger a real ai stack deploy" step. After merge:
Push a trivial change under ai/ to trigger the full pipeline
ssh root@192.168.4.32 grep AI_AWS_ACCESS_KEY_ID /volume1/docker/compose-files/ai/ai.env — should show the new key name populated
After ai stack redeploys: docker service inspect ai_litellm --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}' — confirm AWS_ACCESS_KEY_ID has a real (non-empty) value
Confirm mcpo/mcpo-critical show no diff/restart (unrelated to this change)
## Bug
`ai/ai.yaml`'s `litellm` service (as of commit `37ed671a`, "Change AWS keys to use Woodpecker Secrets", pushed directly to `main` ahead of the secrets-migration PR) references:
```yaml
- AWS_ACCESS_KEY_ID=${AI_AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AI_AWS_SECRET_ACCESS_KEY}
```
i.e. it expects the **AI_-prefixed** host var names, renamed to plain `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` inside the container.
The `ai)` provisioning case in `deploy.yml` (added in the earlier AI secrets migration) wrote the **plain, unprefixed** names into `ai/ai.env` instead. Every other migrated var in `ai.yaml` (`LITELLM_MASTER_KEY`, `DATABASE_URL`, `WEBUI_SECRET_KEY`, etc.) uses plain names — only these two AWS vars use the `AI_` prefix — so this mismatch wasn't caught by pattern-matching against the rest of the file.
## Impact if unfixed
On the next deploy of the `ai` stack, `docker compose config` would render `${AI_AWS_ACCESS_KEY_ID}` as an empty string (since only the plain-named var would exist in the exported env), silently breaking Bedrock/AWS auth in `litellm` — no error, just empty credentials passed to boto3.
## Fix
Changed both the `grep -vE` exclusion pattern and the two `printf` lines in the `ai)` case to use `AI_AWS_ACCESS_KEY_ID` / `AI_AWS_SECRET_ACCESS_KEY` instead of the plain names. Added an inline comment explaining why these two vars specifically need the prefix while the rest don't, to prevent this from silently regressing again in a future rewrite.
## Scope
Only the `ai)` case's two AWS printf lines + the exclusion regex changed. No other stacks, no other AI vars affected.
## Verification after merge
This is the fix needed before the pending "trigger a real ai stack deploy" step. After merge:
1. Push a trivial change under `ai/` to trigger the full pipeline
2. Confirm `provision-secrets` logs `[OK] ai/ai.env secrets updated`
3. `ssh root@192.168.4.32 grep AI_AWS_ACCESS_KEY_ID /volume1/docker/compose-files/ai/ai.env` — should show the new key name populated
4. After `ai` stack redeploys: `docker service inspect ai_litellm --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}'` — confirm `AWS_ACCESS_KEY_ID` has a real (non-empty) value
5. Confirm `mcpo`/`mcpo-critical` show no diff/restart (unrelated to this change)
ai.yaml's litellm service (as of commit 37ed671a, "Change AWS keys to
use Woodpecker Secrets") references ${AI_AWS_ACCESS_KEY_ID} /
${AI_AWS_SECRET_ACCESS_KEY} and renders them into the container as
plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. The ai) provisioning
case added in the earlier secrets-migration PR wrote the plain
(unprefixed) names into ai.env instead, which would leave
${AI_AWS_ACCESS_KEY_ID} unresolved at compose-render time (renders
empty) -- silently breaking Bedrock auth in litellm on the next ai
stack deploy.
Fixed both the grep -vE exclusion pattern and the two printf lines to
use the AI_-prefixed names. All other migrated vars in ai.yaml use
plain names and are unaffected.
No other changes in this file.
AVB
merged commit d1986678bc into main2026-08-25 23:45:28 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Bug
ai/ai.yaml'slitellmservice (as of commit37ed671a, "Change AWS keys to use Woodpecker Secrets", pushed directly tomainahead of the secrets-migration PR) references:i.e. it expects the AI_-prefixed host var names, renamed to plain
AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEYinside the container.The
ai)provisioning case indeploy.yml(added in the earlier AI secrets migration) wrote the plain, unprefixed names intoai/ai.envinstead. Every other migrated var inai.yaml(LITELLM_MASTER_KEY,DATABASE_URL,WEBUI_SECRET_KEY, etc.) uses plain names — only these two AWS vars use theAI_prefix — so this mismatch wasn't caught by pattern-matching against the rest of the file.Impact if unfixed
On the next deploy of the
aistack,docker compose configwould render${AI_AWS_ACCESS_KEY_ID}as an empty string (since only the plain-named var would exist in the exported env), silently breaking Bedrock/AWS auth inlitellm— no error, just empty credentials passed to boto3.Fix
Changed both the
grep -vEexclusion pattern and the twoprintflines in theai)case to useAI_AWS_ACCESS_KEY_ID/AI_AWS_SECRET_ACCESS_KEYinstead of the plain names. Added an inline comment explaining why these two vars specifically need the prefix while the rest don't, to prevent this from silently regressing again in a future rewrite.Scope
Only the
ai)case's two AWS printf lines + the exclusion regex changed. No other stacks, no other AI vars affected.Verification after merge
This is the fix needed before the pending "trigger a real ai stack deploy" step. After merge:
ai/to trigger the full pipelineprovision-secretslogs[OK] ai/ai.env secrets updatedssh root@192.168.4.32 grep AI_AWS_ACCESS_KEY_ID /volume1/docker/compose-files/ai/ai.env— should show the new key name populatedaistack redeploys:docker service inspect ai_litellm --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}'— confirmAWS_ACCESS_KEY_IDhas a real (non-empty) valuemcpo/mcpo-criticalshow no diff/restart (unrelated to this change)