Rebase of the provisioning-v2 branch's deploy.yml onto current main.
Reconstructed from main line-by-line, then two scoped changes:
1. ai) case -> `python3 deploy/provision-stack.py ai` (one line). Retires
main's current heredoc, which has active drift found during this
rebase: AI_LITELLM_POSTGRES_PASSWORD printed TWICE (first copy sourced
from the retired AI_LITELLM_DB_PASSWORD var), a stray legacy
AI_WEBUI_SECRET_KEY key current ai.yaml never reads, and printf
referencing AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET while the env block
declares AI_OPEN_WEB_UI_OAUTH_CLIENT_SECRET (WEB_UI) — undefined shell
var at runtime, so rendered ai.env currently carries an EMPTY OAuth
client secret.
2. env block ai section: normalized to the exact template placeholder
names (AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET <- ai_oauth_client_secret,
AI_OPEN_WEBUI_SECRET_KEY <- ai_webui_secret_key), added AI_MCPO_API_KEY
(NEW Woodpecker secret required pre-merge), pruned now-template-literal
vars (AI_AWS_REGION_NAME, AI_LITELLM_MODIFY_PARAMS/_DATABASE_MIGRATIONS,
AI_LITELLM_DB_PASSWORD, the seven ai_open_web_ui_* config secrets).
apk line gains python3 py3-yaml. All other steps and cases byte-match
current main. Dollar-escape audit done per header lesson.
Key names now match exactly what current ai/ai.yaml references:
AI_OPEN_WEBUI_* (was WEBUI_*/OAUTH_*), AI_LITELLM_* (was LITELLM_*/
DATABASE_URL/POSTGRES_PASSWORD), AI_AWS_REGION_NAME (was AWS_REGION_NAME).
MCPO_API_KEY stays unprefixed (ai.yaml references it unprefixed for both
mcpo and mcpo-critical).
Placeholder names match the CI env vars declared in deploy.yml's
provision-secrets block, including the normalized
AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET (fixing main's WEB_UI/WEBUI typo that
currently renders an EMPTY OAuth client secret into ai.env).
Part 4/4 of the provisioning redesign (matched with secrets-map.yaml,
ai/ai.env.template, deploy/provision-stack.py in this same PR).
Changes, all scoped to the provision-secrets step:
- ai) case: the 54-line grep-v+printf heredoc (which had accumulated a
missing '=' on the DATABASE_URL printf, duplicate POSTGRES_PASSWORD and
WEBUI_SECRET_KEY printfs, and a commented-out line from manual edits —
the 2026-09-03 outage) is replaced by:
python3 deploy/provision-stack.py ai
No secret value structure (connection-string shapes etc.) remains in
this file for the ai stack.
- environment block: pruned the AI_* declarations the template no longer
needs (region/oauth-config/litellm-boolean values are now literals in
ai/ai.env.template): AI_LITELLM_DB_PASSWORD, AI_LITELLM_DATABASE_
MIGRATIONS, AI_LITELLM_MODIFY_PARAMS, AI_AWS_REGION_NAME, and the
seven AI_OPEN_WEB_UI_* config entries. Added AI_MCPO_API_KEY
(from_secret: ai_mcpo_api_key — NEW Woodpecker secret, must exist
before merge). Kept the 9 real ai secrets + 3 flowagent_* entries.
- apk line gains python3 py3-yaml for the provisioner.
- Header comment: documented the 2026-09-03 incident + migration model.
Everything else — validate, all other stacks' case entries, deploy,
verify, notify steps — is byte-for-byte identical to main.
Dollar-escape audit done per the header's own lesson: all remaining
double-dollar-brace references belong to unmigrated legacy cases and are
unchanged; the new ai) case contains no dollar sequences at all.
Part 3/4 of the provisioning redesign. One script for every migrated
stack; no per-stack shell code.
- Renders the stack's full env file from its .env.template (whole file,
every run — no line surgery), fails hard listing NAMES of any
missing/empty vars, fails hard on unresolved placeholders.
- Ships the rendered file over ssh stdin with write-temp + chmod 600 +
atomic mv. Secret values never appear on any command line (an
improvement over create-secrets.sh, which passes values as remote
shell arguments).
- Creates/rotates Docker Swarm secrets from the manifest's
docker_secrets map, values via ssh stdin, same sha256-checksum-label
skip-if-unchanged convention as create-secrets.sh.
- Never prints a secret value — names and counts only.
- Stacks absent from the manifest exit 0 (legacy case-entries keep
handling them), enabling one-stack-at-a-time migration.
Part 2/4 of the provisioning redesign. Key properties:
- Complete key list for ai.env in one reviewable place. Rendered whole
every run by provision-stack.py — the "grep -v + printf line surgery"
that dropped MCPO_API_KEY/AWS_REGION_NAME/OAUTH_* keys (2026-09-03
incident) is gone for this stack.
- Non-secret config (region, OAuth endpoints/IDs/scopes, WEBUI_URL,
LiteLLM booleans) as literals, values taken from the verified-working
host ai.env. Secrets as placeholders resolved from Woodpecker secrets;
renderer fails hard on any missing/empty placeholder.
- LiteLLM DATABASE_URL comes from ai_litellm_database_url as a complete
opaque value — no connection-string structure in git (addresses the
plaintext-structure concern in deploy.yml).
- Drops legacy plain AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY keys —
ai.yaml only references the AI_-prefixed names.
- NEW Woodpecker secret required before merging: ai_mcpo_api_key
(value = current MCPO_API_KEY from host ai.env).
Part 1/4 of the provisioning architecture redesign after the 2026-09-03
ai.env incident (line-surgery provisioning dropped keys; broken services;
manual host-side recovery forked ai.yaml/ai.env).
Design: plain-data manifest consumed by deploy/provision-stack.py.
No code, no shell, no secret values, no value structure (e.g. no
connection-string shapes) live in this file or in deploy.yml anymore.
Bot