Rebase of the provisioning-v2 branch's deploy.yml onto current main.
Reconstructed from main line-by-line, then two scoped changes:
1. ai) case -> `python3 deploy/provision-stack.py ai` (one line). Retires
main's current heredoc, which has active drift found during this
rebase: AI_LITELLM_POSTGRES_PASSWORD printed TWICE (first copy sourced
from the retired AI_LITELLM_DB_PASSWORD var), a stray legacy
AI_WEBUI_SECRET_KEY key current ai.yaml never reads, and printf
referencing AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET while the env block
declares AI_OPEN_WEB_UI_OAUTH_CLIENT_SECRET (WEB_UI) — undefined shell
var at runtime, so rendered ai.env currently carries an EMPTY OAuth
client secret.
2. env block ai section: normalized to the exact template placeholder
names (AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET <- ai_oauth_client_secret,
AI_OPEN_WEBUI_SECRET_KEY <- ai_webui_secret_key), added AI_MCPO_API_KEY
(NEW Woodpecker secret required pre-merge), pruned now-template-literal
vars (AI_AWS_REGION_NAME, AI_LITELLM_MODIFY_PARAMS/_DATABASE_MIGRATIONS,
AI_LITELLM_DB_PASSWORD, the seven ai_open_web_ui_* config secrets).
apk line gains python3 py3-yaml. All other steps and cases byte-match
current main. Dollar-escape audit done per header lesson.
Part 4/4 of the provisioning redesign (matched with secrets-map.yaml,
ai/ai.env.template, deploy/provision-stack.py in this same PR).
Changes, all scoped to the provision-secrets step:
- ai) case: the 54-line grep-v+printf heredoc (which had accumulated a
missing '=' on the DATABASE_URL printf, duplicate POSTGRES_PASSWORD and
WEBUI_SECRET_KEY printfs, and a commented-out line from manual edits —
the 2026-09-03 outage) is replaced by:
python3 deploy/provision-stack.py ai
No secret value structure (connection-string shapes etc.) remains in
this file for the ai stack.
- environment block: pruned the AI_* declarations the template no longer
needs (region/oauth-config/litellm-boolean values are now literals in
ai/ai.env.template): AI_LITELLM_DB_PASSWORD, AI_LITELLM_DATABASE_
MIGRATIONS, AI_LITELLM_MODIFY_PARAMS, AI_AWS_REGION_NAME, and the
seven AI_OPEN_WEB_UI_* config entries. Added AI_MCPO_API_KEY
(from_secret: ai_mcpo_api_key — NEW Woodpecker secret, must exist
before merge). Kept the 9 real ai secrets + 3 flowagent_* entries.
- apk line gains python3 py3-yaml for the provisioner.
- Header comment: documented the 2026-09-03 incident + migration model.
Everything else — validate, all other stacks' case entries, deploy,
verify, notify steps — is byte-for-byte identical to main.
Dollar-escape audit done per the header's own lesson: all remaining
double-dollar-brace references belong to unmigrated legacy cases and are
unchanged; the new ai) case contains no dollar sequences at all.
Surgical addition to the existing case "ai)" block only — every other
case, comment, and line in this ~26KB pipeline file is byte-for-byte
unchanged from main. Given this file's documented incident history from
prior full-file rewrites (see header comment), this was hand-verified
against the current main content line-by-line before writing, not
generated fresh.
Adds:
- 3 new environment: entries (FLOWAGENT_AZURE_CLIENT_ID/TENANT_ID/
CLIENT_SECRET) via from_secret, alongside the existing ai_* secrets
- Inside the existing `ai)` case, after the ai.env rewrite block: three
create_or_update_secret calls provisioning flowagent_azure_client_id,
flowagent_azure_tenant_id, flowagent_azure_client_secret as native
Docker Swarm secrets (Pattern C, _FILE convention) — NOT written into
ai/ai.env, since mcpo's flowagent entry reads them via
/run/secrets/flowagent_azure_* per flowagent/entrypoint.sh, not env vars
- A short comment explaining why these three are Pattern C secrets
inside an otherwise Pattern-B (host .env) stack case
This is the matched-pair companion to the ai.yaml commit in this same PR,
per the FlowAgent progress note's explicit sequencing requirement (both
land together, reviewed, not auto-merged to main).
Context: git hosts the source of truth for every other stack's compose
files, so in a disaster-recovery scenario it must be restorable from a
flat git.yaml + git.env backup alone, with zero dependency on a running
Swarm's Docker secret store (native Docker secrets can't be resolved
until Swarm already exists, which is the circular dependency git.yaml
would otherwise create). Path 2 (documented decision): Pattern B for
git specifically, same mechanism the ai stack already uses.
provision-secrets / git) case:
- Previously created native Docker secrets (git_db_password,
git_runner_token, git_mcp_access_token) via create-secrets.sh —
leftover from an earlier, abandoned Pattern C attempt.
- Now rewrites only the secret-bearing lines (GITEA__database__PASSWD,
GITEA_RUNNER_REGISTRATION_TOKEN, GITEA_MCP_ACCESS_TOKEN) via
grep -v + printf, mirroring the ai) case exactly. Also strips the
legacy GITEA_ACCESS_TOKEN key name so the test file converges on the
git.env.example-documented key.
- TEST PHASE: target is git.env.pipelinetest, NOT git.env. The real
git.env is never opened for writing by this step. First run seeds
the test file from the real git.env (to carry over all non-secret
lines), then only the 3 secret lines are refreshed on every push.
- Real cutover (pointing git.yaml/stack-deploy at the generated file,
then retiring git.env.pipelinetest) is a deliberate follow-up step
after manually diffing the rendered output.
deploy / bootstrap-tier guard:
- Added git to the traefik|woodpecker|postgresql|secrets guard list.
git.yaml changes now sync to the host but require a manual
`stack-deploy.sh git` run, same as the other foundational stacks —
prevents an auto-deploy of a bad git.yaml change from taking down
Gitea before a human can look at it (Gitea itself is what every
other pipeline needs to trigger a fix).
grep -E '^[^/.][^/]*/' matches ANY non-dot top-level folder in the
changed-files list, including deploy/ -- the shared tooling folder
synced by every deploy, not a stack. A PR touching only
deploy/envparse.py caused stack-deploy.sh to be invoked with "deploy"
as a stack name, which correctly errored ("No main compose file... in
.../deploy") since deploy/ has no deploy.yaml.
No live service was affected (the error occurs before any redeploy
attempt), but it produced a confusing FAIL on an otherwise-correct
change (PR #8) and could mask a real failure in the noise.
Adds `| grep -v '^deploy$'` after the folder-name extraction in all 5
places this pattern appears (validate, provision-secrets x2, deploy,
verify). deploy/ is already unconditionally rsynced at the top of the
deploy step regardless of which stacks changed, so excluding it from
the stack list is safe -- it will still be synced, just never treated
as a deployable stack.
ai.yaml's litellm service (as of commit 37ed671a, "Change AWS keys to
use Woodpecker Secrets") references ${AI_AWS_ACCESS_KEY_ID} /
${AI_AWS_SECRET_ACCESS_KEY} and renders them into the container as
plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. The ai) provisioning
case added in the earlier secrets-migration PR wrote the plain
(unprefixed) names into ai.env instead, which would leave
${AI_AWS_ACCESS_KEY_ID} unresolved at compose-render time (renders
empty) -- silently breaking Bedrock auth in litellm on the next ai
stack deploy.
Fixed both the grep -vE exclusion pattern and the two printf lines to
use the AI_-prefixed names. All other migrated vars in ai.yaml use
plain names and are unaffected.
No other changes in this file.
The AI secrets migration PR (#4) was a full-file rewrite of deploy.yml.
That rewrite mechanically dropped one $ from EVERY $${VAR} occurrence in
the file, not just the new AI additions -- silently reverting all
pre-existing secret references (SWARM_MANAGER_IP, IMMICH_*, GIT_*,
POSTGRESQL_*, VAULTWARDEN_*, ENTERTAINMENT_*, etc.) to single-dollar
form. Per this file's own header comment, Woodpecker blanks single-dollar
braced refs at compile time since secrets aren't in that variable map --
this is the exact "SWARM_MANAGER_IP secret is empty" failure mode
documented above, and it fired immediately on the first push after #4
merged.
Impact: provision-secrets/deploy/verify all failed at their first
if-empty guard and exited before any ssh/scp/rsync ran. No live secret,
service, or deployed stack was touched -- this was a CI-only outage.
Fix: restored $${VAR} for every secret-backed reference throughout the
file. CI_PIPELINE_FILES / CI_COMMIT_MESSAGE stay single-dollar (correct
-- those are Woodpecker compile-time metadata, not secrets). The \$FILE
/ \$TMP backslash-escaping inside the ai) case's remote SSH command is
unrelated and was already correct (it protects those local-to-remote
vars from expanding before the SSH payload is sent).
This is a straight revert-of-the-regression -- no new secrets, no logic
changes beyond restoring the escaping.
Adds 8 new from_secret-backed env vars to provision-secrets and rewrites
the `ai)` case to do a targeted update of only those 8 keys in the
remote ai/ai.env via grep -v + printf (no sed, safe for values containing
/, $, &, etc). All other lines in ai.env (MCPO_API_KEY, OAUTH_CLIENT_ID,
WEBUI_URL, etc.) are left completely untouched -- MCPO_API_KEY migration
is deferred to a follow-up per plan, and this change never reads or
writes that value.
New secrets required in Woodpecker (Settings -> Secrets) before merge:
ai_aws_access_key_id
ai_aws_secret_access_key
ai_litellm_master_key
ai_litellm_salt_key
ai_litellm_db_password
ai_webui_secret_key
ai_open_webui_database_url
ai_oauth_client_secret
Root cause of Woodpecker push pipelines silently no-op'ing since
2026-07-18: Woodpecker's config resolution is
.woodpecker/*.{yaml,yml} -> .woodpecker.yaml -> .woodpecker.yml
Once .woodpecker/renovate.yml existed, the root .woodpecker.yml was
entirely ignored. Every push webhook hit renovate.yml's `when: event:
cron` filter, matched zero steps, and Woodpecker logged
"ignoring hook: 'when' filters filtered out all steps" instead of
running deploy.
Moving the push pipeline into .woodpecker/deploy.yml restores it as
an independent workflow alongside renovate.yml -- each file's own
`when:` filter now applies correctly (push -> deploy.yml, cron ->
renovate.yml).
Bot
admin
AVB