Key names now match exactly what current ai/ai.yaml references:
AI_OPEN_WEBUI_* (was WEBUI_*/OAUTH_*), AI_LITELLM_* (was LITELLM_*/
DATABASE_URL/POSTGRES_PASSWORD), AI_AWS_REGION_NAME (was AWS_REGION_NAME).
MCPO_API_KEY stays unprefixed (ai.yaml references it unprefixed for both
mcpo and mcpo-critical).
Placeholder names match the CI env vars declared in deploy.yml's
provision-secrets block, including the normalized
AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET (fixing main's WEB_UI/WEBUI typo that
currently renders an EMPTY OAuth client secret into ai.env).
Part 4/4 of the provisioning redesign (matched with secrets-map.yaml,
ai/ai.env.template, deploy/provision-stack.py in this same PR).
Changes, all scoped to the provision-secrets step:
- ai) case: the 54-line grep-v+printf heredoc (which had accumulated a
missing '=' on the DATABASE_URL printf, duplicate POSTGRES_PASSWORD and
WEBUI_SECRET_KEY printfs, and a commented-out line from manual edits —
the 2026-09-03 outage) is replaced by:
python3 deploy/provision-stack.py ai
No secret value structure (connection-string shapes etc.) remains in
this file for the ai stack.
- environment block: pruned the AI_* declarations the template no longer
needs (region/oauth-config/litellm-boolean values are now literals in
ai/ai.env.template): AI_LITELLM_DB_PASSWORD, AI_LITELLM_DATABASE_
MIGRATIONS, AI_LITELLM_MODIFY_PARAMS, AI_AWS_REGION_NAME, and the
seven AI_OPEN_WEB_UI_* config entries. Added AI_MCPO_API_KEY
(from_secret: ai_mcpo_api_key — NEW Woodpecker secret, must exist
before merge). Kept the 9 real ai secrets + 3 flowagent_* entries.
- apk line gains python3 py3-yaml for the provisioner.
- Header comment: documented the 2026-09-03 incident + migration model.
Everything else — validate, all other stacks' case entries, deploy,
verify, notify steps — is byte-for-byte identical to main.
Dollar-escape audit done per the header's own lesson: all remaining
double-dollar-brace references belong to unmigrated legacy cases and are
unchanged; the new ai) case contains no dollar sequences at all.
Part 3/4 of the provisioning redesign. One script for every migrated
stack; no per-stack shell code.
- Renders the stack's full env file from its .env.template (whole file,
every run — no line surgery), fails hard listing NAMES of any
missing/empty vars, fails hard on unresolved placeholders.
- Ships the rendered file over ssh stdin with write-temp + chmod 600 +
atomic mv. Secret values never appear on any command line (an
improvement over create-secrets.sh, which passes values as remote
shell arguments).
- Creates/rotates Docker Swarm secrets from the manifest's
docker_secrets map, values via ssh stdin, same sha256-checksum-label
skip-if-unchanged convention as create-secrets.sh.
- Never prints a secret value — names and counts only.
- Stacks absent from the manifest exit 0 (legacy case-entries keep
handling them), enabling one-stack-at-a-time migration.
Part 2/4 of the provisioning redesign. Key properties:
- Complete key list for ai.env in one reviewable place. Rendered whole
every run by provision-stack.py — the "grep -v + printf line surgery"
that dropped MCPO_API_KEY/AWS_REGION_NAME/OAUTH_* keys (2026-09-03
incident) is gone for this stack.
- Non-secret config (region, OAuth endpoints/IDs/scopes, WEBUI_URL,
LiteLLM booleans) as literals, values taken from the verified-working
host ai.env. Secrets as placeholders resolved from Woodpecker secrets;
renderer fails hard on any missing/empty placeholder.
- LiteLLM DATABASE_URL comes from ai_litellm_database_url as a complete
opaque value — no connection-string structure in git (addresses the
plaintext-structure concern in deploy.yml).
- Drops legacy plain AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY keys —
ai.yaml only references the AI_-prefixed names.
- NEW Woodpecker secret required before merging: ai_mcpo_api_key
(value = current MCPO_API_KEY from host ai.env).
Part 1/4 of the provisioning architecture redesign after the 2026-09-03
ai.env incident (line-surgery provisioning dropped keys; broken services;
manual host-side recovery forked ai.yaml/ai.env).
Design: plain-data manifest consumed by deploy/provision-stack.py.
No code, no shell, no secret values, no value structure (e.g. no
connection-string shapes) live in this file or in deploy.yml anymore.
Surgical addition to the existing case "ai)" block only — every other
case, comment, and line in this ~26KB pipeline file is byte-for-byte
unchanged from main. Given this file's documented incident history from
prior full-file rewrites (see header comment), this was hand-verified
against the current main content line-by-line before writing, not
generated fresh.
Adds:
- 3 new environment: entries (FLOWAGENT_AZURE_CLIENT_ID/TENANT_ID/
CLIENT_SECRET) via from_secret, alongside the existing ai_* secrets
- Inside the existing `ai)` case, after the ai.env rewrite block: three
create_or_update_secret calls provisioning flowagent_azure_client_id,
flowagent_azure_tenant_id, flowagent_azure_client_secret as native
Docker Swarm secrets (Pattern C, _FILE convention) — NOT written into
ai/ai.env, since mcpo's flowagent entry reads them via
/run/secrets/flowagent_azure_* per flowagent/entrypoint.sh, not env vars
- A short comment explaining why these three are Pattern C secrets
inside an otherwise Pattern-B (host .env) stack case
This is the matched-pair companion to the ai.yaml commit in this same PR,
per the FlowAgent progress note's explicit sequencing requirement (both
land together, reviewed, not auto-merged to main).
Image pinned to git.bryanmail.net/homelab/flowagent-mcpo:916164714429 —
the exact sha-tagged image built and smoke-tested clean in mcp-config
pipeline #22 (build-and-push-flowagent + smoke-test-flowagent-image both
success). NOT :latest, for reproducible deploys.
Adds a secrets: block to the mcpo service only, referencing the 3
flowagent_azure_* Docker Swarm secrets (Pattern C, _FILE convention,
consumed by flowagent/entrypoint.sh via /run/secrets/flowagent_azure_*).
Declares those 3 secrets as external at top level, mirroring the existing
vaultwarden_admin_token_v2 pattern in vaultwarden.yaml.
Companion secret-provisioning change lands in the same PR as a separate
commit in .woodpecker/deploy.yml's provision-secrets step (case "ai)").
mcpo-critical (lines defining Proxmox access) is completely untouched —
still stock ghcr.io/open-webui/mcpo:main, no secrets: block, unchanged.
Per FlowAgent MCP progress note: Azure AD App Registration is reported
created with real secrets already in Woodpecker (both mcp-config and
compose-files repos) as of this PR. This is a reviewed PR against a
feature branch, NOT auto-merged to main — deploy only happens after
manual review/merge, per user's explicit request for a manual-revert
safety net on this high-blast-radius shared stack (LiteLLM, Open WebUI,
n8n also live here).
Context: git hosts the source of truth for every other stack's compose
files, so in a disaster-recovery scenario it must be restorable from a
flat git.yaml + git.env backup alone, with zero dependency on a running
Swarm's Docker secret store (native Docker secrets can't be resolved
until Swarm already exists, which is the circular dependency git.yaml
would otherwise create). Path 2 (documented decision): Pattern B for
git specifically, same mechanism the ai stack already uses.
provision-secrets / git) case:
- Previously created native Docker secrets (git_db_password,
git_runner_token, git_mcp_access_token) via create-secrets.sh —
leftover from an earlier, abandoned Pattern C attempt.
- Now rewrites only the secret-bearing lines (GITEA__database__PASSWD,
GITEA_RUNNER_REGISTRATION_TOKEN, GITEA_MCP_ACCESS_TOKEN) via
grep -v + printf, mirroring the ai) case exactly. Also strips the
legacy GITEA_ACCESS_TOKEN key name so the test file converges on the
git.env.example-documented key.
- TEST PHASE: target is git.env.pipelinetest, NOT git.env. The real
git.env is never opened for writing by this step. First run seeds
the test file from the real git.env (to carry over all non-secret
lines), then only the 3 secret lines are refreshed on every push.
- Real cutover (pointing git.yaml/stack-deploy at the generated file,
then retiring git.env.pipelinetest) is a deliberate follow-up step
after manually diffing the rendered output.
deploy / bootstrap-tier guard:
- Added git to the traefik|woodpecker|postgresql|secrets guard list.
git.yaml changes now sync to the host but require a manual
`stack-deploy.sh git` run, same as the other foundational stacks —
prevents an auto-deploy of a bad git.yaml change from taking down
Gitea before a human can look at it (Gitea itself is what every
other pipeline needs to trigger a fix).
Bot