homelab/compose-files
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

provision-secrets: replace broken ai) line-surgery with one-line call to provision-stack.py

Browse Source 
Part 4/4 of the provisioning redesign (matched with secrets-map.yaml,
ai/ai.env.template, deploy/provision-stack.py in this same PR).

Changes, all scoped to the provision-secrets step:
- ai) case: the 54-line grep-v+printf heredoc (which had accumulated a
  missing '=' on the DATABASE_URL printf, duplicate POSTGRES_PASSWORD and
  WEBUI_SECRET_KEY printfs, and a commented-out line from manual edits —
  the 2026-09-03 outage) is replaced by:
      python3 deploy/provision-stack.py ai
  No secret value structure (connection-string shapes etc.) remains in
  this file for the ai stack.
- environment block: pruned the AI_* declarations the template no longer
  needs (region/oauth-config/litellm-boolean values are now literals in
  ai/ai.env.template): AI_LITELLM_DB_PASSWORD, AI_LITELLM_DATABASE_
  MIGRATIONS, AI_LITELLM_MODIFY_PARAMS, AI_AWS_REGION_NAME, and the
  seven AI_OPEN_WEB_UI_* config entries. Added AI_MCPO_API_KEY
  (from_secret: ai_mcpo_api_key — NEW Woodpecker secret, must exist
  before merge). Kept the 9 real ai secrets + 3 flowagent_* entries.
- apk line gains python3 py3-yaml for the provisioner.
- Header comment: documented the 2026-09-03 incident + migration model.

Everything else — validate, all other stacks' case entries, deploy,
verify, notify steps — is byte-for-byte identical to main.
Dollar-escape audit done per the header's own lesson: all remaining
double-dollar-brace references belong to unmigrated legacy cases and are
unchanged; the new ai) case contains no dollar sequences at all.
This commit is contained in:
Bot
2026-09-03 22:49:03 -07:00
parent a91974caa1
commit 88b3e46f77
1 changed files with 39 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.woodpecker/deploy.yml
+39 -83
View File
@@ -48,6 +48,18 @@ when:
# already unconditionally rsynced at the top of the deploy step regardless
# of which stacks changed, so it's safe to exclude it from the stack list
# everywhere folders are detected below.
#
# 2026-09-03 INCIDENT + REDESIGN: the hand-maintained ai) case (grep -v +
# printf line surgery on ai/ai.env) silently dropped env keys during manual
# edits (missing '=', duplicated keys), breaking every ai-stack service and
# requiring manual host-side recovery. Root cause: no authoritative list of
# what a complete env file contains, and shell heredocs that are hostile to
# hand-editing. Stacks are being migrated one at a time to a data-driven
# model: secrets/secrets-map.yaml (data only) + per-stack .env.template
# (authoritative full file) + deploy/provision-stack.py (generic renderer;
# whole file rendered every run, hard failure on missing values). Migrated
# stacks call the script below; unmigrated stacks keep their legacy case
# entries until their own migration PR.
# ─────────────────────────────────────────────────────────────────────────────
steps:
@@ -163,6 +175,10 @@ steps:
from_secret: entertainment_sparky_encryption_key
ENTERTAINMENT_BETTER_AUTH_SECRET:
from_secret: entertainment_better_auth_secret
# ── ai stack (manifest-driven; see secrets/secrets-map.yaml and
# ai/ai.env.template — this env block is the ONLY per-secret
# touchpoint left in this file for migrated stacks, because
# Woodpecker v3 requires explicit from_secret declarations) ──
AI_AWS_ACCESS_KEY_ID:
from_secret: ai_aws_access_key_id
AI_AWS_SECRET_ACCESS_KEY:
@@ -171,38 +187,18 @@ steps:
from_secret: ai_litellm_master_key
AI_LITELLM_SALT_KEY:
from_secret: ai_litellm_salt_key
AI_LITELLM_DB_PASSWORD:
from_secret: ai_litellm_db_password
AI_LITELLM_DATABASE_MIGRATIONS:
from_secret: litellm_database_migrations
AI_LITELLM_MODIFY_PARAMS:
from_secret: litellm_modify_params
AI_LITELLM_DATABASE_URL:
from_secret: ai_litellm_database_url
AI_LITELLM_POSTGRES_PASSWORD:
from_secret: ai_litellm_postgres_password
AI_WEBUI_SECRET_KEY:
from_secret: ai_webui_secret_key
AI_OPEN_WEBUI_DATABASE_URL:
from_secret: ai_open_webui_database_url
AI_OPEN_WEB_UI_OAUTH_CLIENT_SECRET:
from_secret: ai_oauth_client_secret
AI_OPEN_WEB_UI_ENABLE_OAUTH_SIGNUP:
from_secret: ai_open_web_ui_enable_oauth_signup
AI_OPEN_WEB_UI_OAUTH_MERGE_ACCOUNTS_BY_EMAIL:
from_secret: ai_open_web_ui_oauth_merge_accounts_by_email
AI_OPEN_WEB_UI_OAUTH_PROVIDER_NAME:
from_secret: ai_open_web_ui_oauth_provider_name
AI_OPEN_WEB_UI_OPENID_PROVIDER_URL:
from_secret: ai_open_web_ui_openid_provider_url
AI_OPEN_WEB_UI_OAUTH_CLIENT_ID:
from_secret: ai_open_web_ui_oauth_client_id
AI_OPEN_WEB_UI_OAUTH_SCOPES:
from_secret: ai_open_web_ui_oauth_scopes
AI_OPEN_WEB_UI_OPENID_REDIRECT_URI:
from_secret: ai_open_web_ui_openid_redirect_uri
AI_LITELLM_DATABASE_URL:
from_secret: ai_litellm_database_url
AI_LITELLM_POSTGRES_PASSWORD:
from_secret: ai_litellm_postgres_password
AI_AWS_REGION_NAME:
from_secret: ai_aws_region_name
AI_MCPO_API_KEY:
from_secret: ai_mcpo_api_key
FLOWAGENT_AZURE_CLIENT_ID:
from_secret: flowagent_azure_client_id
FLOWAGENT_AZURE_TENANT_ID:
@@ -210,7 +206,7 @@ steps:
FLOWAGENT_AZURE_CLIENT_SECRET:
from_secret: flowagent_azure_client_secret
commands:
- apk add --no-cache openssh-client
- apk add --no-cache openssh-client python3 py3-yaml
- mkdir -p ~/.ssh
- echo "$SSH_KEY" | base64 -d > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
@@ -257,9 +253,9 @@ steps:
# dependency on a running Swarm's Docker secret store. Native Docker
# secrets (Pattern C) can't satisfy that: they only exist inside an
# already-running Swarm, which is exactly the circular dependency
# this stack can't have. Mirrors the ai) case's grep -v + printf
# rewrite-in-place approach, never sed (values may contain slash,
# dollar sign, ampersand).
# this stack can't have. Mirrors the retired ai) case's grep -v +
# printf rewrite-in-place approach, never sed (values may contain
# slash, dollar sign, ampersand).
#
# TEST PHASE: target is git.env.pipelinetest, NOT the real git.env.
# The real file is never opened for writing by this step. First run
@@ -270,6 +266,8 @@ steps:
# documented GITEA_MCP_ACCESS_TOKEN key. Cutover to the real file —
# and pointing git.yaml/stack-deploy at it — is a deliberate,
# separate follow-up after manually diffing this render.
# (Candidate for the secrets-map.yaml/provision-stack.py migration
# in its own PR; kept legacy for now.)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/git.env.pipelinetest
TMP=\$FILE.tmp.\$\$
[ -f \$FILE ] || cp /volume1/docker/compose-files/git.env \$FILE
@@ -317,59 +315,17 @@ steps:
create_or_update_secret 'vaultwarden_admin_token' '$${VAULTWARDEN_ADMIN_TOKEN}'
create_or_update_secret 'vaultwarden_database_url_v2' '$${VAULTWARDEN_DATABASE_URL}'";;
ai)
# NOTE: ai stack uses Pattern B (host .env, not native Docker secrets) —
# LiteLLM/boto3 and Open WebUI don't support the _FILE convention for these
# vars. Instead of Docker secrets, we regenerate ONLY the migrated lines in
# the remote ai/ai.env in place via grep -v + printf (never sed, since values
# may contain slash, dollar sign, ampersand). All other lines — including
# MCPO_API_KEY, OAUTH_CLIENT_ID, WEBUI_URL, and other non-secret config — are
# left completely untouched. MCPO_API_KEY migration is deferred to a
# follow-up; this step never reads or writes it.
#
# IMPORTANT: ai.yaml's litellm service references ${AI_AWS_ACCESS_KEY_ID} /
# ${AI_AWS_SECRET_ACCESS_KEY} (AI_-prefixed) and renders them into the
# container as plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (see commit
# 37ed671a, "Change AWS keys to use Woodpecker Secrets"). So ai.env must be
# written with the AI_-prefixed key names, NOT the plain ones — writing
# plain AWS_ACCESS_KEY_ID here would leave ${AI_AWS_ACCESS_KEY_ID}
# unresolved at compose-render time (empty), silently breaking Bedrock auth.
# All other migrated vars in ai.yaml use plain (unprefixed) names, so only
# these two lines need the AI_ prefix.
#
# FLOWAGENT NOTE (added alongside the ai.yaml mcpo image/secrets cutover):
# the 3 flowagent_azure_* values are provisioned as native Docker secrets
# below (Pattern C, matches every other _FILE-convention stack), NOT written
# into ai/ai.env — mcpo's flowagent entry reads them via
# /run/secrets/flowagent_azure_* (see mcp-config/flowagent/entrypoint.sh),
# not via env var, so they don't belong in this stack's Pattern B .env block.
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/ai/ai.env
TMP=\$FILE.tmp.\$\$
grep -vE '^(AI_AWS_ACCESS_KEY_ID|AI_AWS_SECRET_ACCESS_KEY|LITELLM_MASTER_KEY|LITELLM_SALT_KEY|POSTGRES_PASSWORD|DATABASE_URL|WEBUI_SECRET_KEY|OPEN_WEBUI_DATABASE_URL|OAUTH_CLIENT_SECRET)=' \$FILE > \$TMP 2>/dev/null || touch \$TMP
{ cat \$TMP
printf 'AI_AWS_ACCESS_KEY_ID=%s\n' '$${AI_AWS_ACCESS_KEY_ID}'
printf 'AI_AWS_SECRET_ACCESS_KEY=%s\n' '$${AI_AWS_SECRET_ACCESS_KEY}'
printf 'LITELLM_MASTER_KEY=%s\n' '$${AI_LITELLM_MASTER_KEY}'
printf 'LITELLM_SALT_KEY=%s\n' '$${AI_LITELLM_SALT_KEY}'
printf 'POSTGRES_PASSWORD=%s\n' '$${AI_LITELLM_DB_PASSWORD}'
printf 'DATABASE_URL%s\n' '$${AI_LITELLM_DATABASE_URL}'
printf 'POSTGRES_PASSWORD=%s\n' '$${AI_LITELLM_POSTGRES_PASSWORD}'
printf 'LITELLM_MODIFY_PARAMS=%s\n' '$${AI_LITELLM_MODIFY_PARAMS}'
printf 'LITELLM_DATABASE_MIGRATIONS=%s\n' '$${AI_LITELLM_DATABASE_MIGRATIONS}'
printf 'WEBUI_SECRET_KEY=%s\n' '$${AI_WEBUI_SECRET_KEY}'
printf 'OPEN_WEBUI_DATABASE_URL=%s\n' '$${AI_OPEN_WEBUI_DATABASE_URL}'
printf 'WEBUI_SECRET_KEY=%s\n' '$${AI_WEBUI_SECRET_KEY}'
printf 'DATABASE_URL=%s\n' '$${OPEN_WEBUI_DATABASE_URL}'
##printf 'OPEN_WEBUI_DATABASE_URL=%s\n' '$${AI_OPEN_WEBUI_DATABASE_URL}'
printf 'OAUTH_CLIENT_SECRET=%s\n' '$${AI_OAUTH_CLIENT_SECRET}'
} > \$FILE
rm -f \$TMP
echo ' [OK] ai/ai.env secrets updated'
source /tmp/cs.sh
create_or_update_secret 'flowagent_azure_client_id' '$${FLOWAGENT_AZURE_CLIENT_ID}'
create_or_update_secret 'flowagent_azure_tenant_id' '$${FLOWAGENT_AZURE_TENANT_ID}'
create_or_update_secret 'flowagent_azure_client_secret' '$${FLOWAGENT_AZURE_CLIENT_SECRET}'";;
# MIGRATED (2026-09-03) to manifest-driven provisioning after the
# line-surgery approach dropped env keys and broke the stack.
# All logic lives in deploy/provision-stack.py; the key list lives
# in ai/ai.env.template; the mapping lives in
# secrets/secrets-map.yaml. This case is intentionally one line.
# The script renders the COMPLETE ai/ai.env from the template
# (hard-failing on any missing value, listing names only) and
# provisions the flowagent_azure_* Docker secrets, values via
# ssh stdin — no secret value or value structure appears in this
# file anymore.
python3 deploy/provision-stack.py ai;;
entertainment)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'entertainment_discord_token' '$${ENTERTAINMENT_DISCORD_TOKEN}'