Comment-only change. Forces a real deploy of the ai stack now that
PR #8 (envparse.py $$ escaping) and PR #9 (deploy/ folder exclusion)
are both merged, so litellm picks up the correctly-escaped
LITELLM_MASTER_KEY/LITELLM_SALT_KEY instead of the truncated values
currently running (truncated at the first literal '$' due to the
envsubst+Compose double-interpolation bug fixed in #8).
grep -E '^[^/.][^/]*/' matches ANY non-dot top-level folder in the
changed-files list, including deploy/ -- the shared tooling folder
synced by every deploy, not a stack. A PR touching only
deploy/envparse.py caused stack-deploy.sh to be invoked with "deploy"
as a stack name, which correctly errored ("No main compose file... in
.../deploy") since deploy/ has no deploy.yaml.
No live service was affected (the error occurs before any redeploy
attempt), but it produced a confusing FAIL on an otherwise-correct
change (PR #8) and could mask a real failure in the noise.
Adds `| grep -v '^deploy$'` after the folder-name extraction in all 5
places this pattern appears (validate, provision-secrets x2, deploy,
verify). deploy/ is already unconditionally rsynced at the top of the
deploy step regardless of which stacks changed, so excluding it from
the stack list is safe -- it will still be synced, just never treated
as a deployable stack.
Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation
incident (2026-08-26): stack-deploy.sh's single-file deploy path is
`envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`.
envsubst embeds the raw secret value into the compose YAML text. If
that value contains a literal '$' followed by word chars, the
resulting YAML now contains what looks like a second variable
reference. `docker stack deploy -c -` runs Compose's own interpolation
pass on that text before creating the service, finds no such env var,
and silently substitutes empty string -- truncating the secret in the
running container with no error.
Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm
container as 73 chars, silently, on a real deploy.
This is not specific to ai -- it affects every Pattern B stack (host
.env + envsubst, not native Docker secrets): maintenance, media,
unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of
them could have a '$'-containing value truncating right now without
detection, since the failure produces no warning.
Fix: escape every literal '$' as '$$' in export/export_merged (which
feed the `eval` that sets envsubst's actual source values), before
envsubst ever sees them. envsubst does not interpret '$' in replacement
text, so the doubled dollar survives envsubst intact; Compose's own
interpolation pass then consumes exactly one level of escaping,
landing on the correct single '$' with no leftover false variable
reference. vars/vars_merged (envsubst's allowlist string, unrelated to
values) are untouched.
NOT deployed/merged yet -- pending review. The currently-running
ai_litellm service still has the truncated keys and needs a fresh
`stack-deploy.sh ai` run after this merges to pick up the corrected
values.
No functional change -- this comment-only edit exists to trigger a real
deploy of the ai stack so provision-secrets' ai) case, and the AI_
var-name fix from PR #7, get exercised end-to-end for the first time.
Documents that MCPO_API_KEY is intentionally still manual/unmigrated.
ai.yaml's litellm service (as of commit 37ed671a, "Change AWS keys to
use Woodpecker Secrets") references ${AI_AWS_ACCESS_KEY_ID} /
${AI_AWS_SECRET_ACCESS_KEY} and renders them into the container as
plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. The ai) provisioning
case added in the earlier secrets-migration PR wrote the plain
(unprefixed) names into ai.env instead, which would leave
${AI_AWS_ACCESS_KEY_ID} unresolved at compose-render time (renders
empty) -- silently breaking Bedrock auth in litellm on the next ai
stack deploy.
Fixed both the grep -vE exclusion pattern and the two printf lines to
use the AI_-prefixed names. All other migrated vars in ai.yaml use
plain names and are unaffected.
No other changes in this file.
The AI secrets migration PR (#4) was a full-file rewrite of deploy.yml.
That rewrite mechanically dropped one $ from EVERY $${VAR} occurrence in
the file, not just the new AI additions -- silently reverting all
pre-existing secret references (SWARM_MANAGER_IP, IMMICH_*, GIT_*,
POSTGRESQL_*, VAULTWARDEN_*, ENTERTAINMENT_*, etc.) to single-dollar
form. Per this file's own header comment, Woodpecker blanks single-dollar
braced refs at compile time since secrets aren't in that variable map --
this is the exact "SWARM_MANAGER_IP secret is empty" failure mode
documented above, and it fired immediately on the first push after #4
merged.
Impact: provision-secrets/deploy/verify all failed at their first
if-empty guard and exited before any ssh/scp/rsync ran. No live secret,
service, or deployed stack was touched -- this was a CI-only outage.
Fix: restored $${VAR} for every secret-backed reference throughout the
file. CI_PIPELINE_FILES / CI_COMMIT_MESSAGE stay single-dollar (correct
-- those are Woodpecker compile-time metadata, not secrets). The \$FILE
/ \$TMP backslash-escaping inside the ai) case's remote SSH command is
unrelated and was already correct (it protects those local-to-remote
vars from expanding before the SSH payload is sent).
This is a straight revert-of-the-regression -- no new secrets, no logic
changes beyond restoring the escaping.
No stack files changed -- this commit exists only to trigger a fresh
Woodpecker pipeline run against current main + current secrets, since
"Restart" on the prior failed run was replaying a stale snapshot from
before the ai_* secrets existed.
Adds 8 new from_secret-backed env vars to provision-secrets and rewrites
the `ai)` case to do a targeted update of only those 8 keys in the
remote ai/ai.env via grep -v + printf (no sed, safe for values containing
/, $, &, etc). All other lines in ai.env (MCPO_API_KEY, OAUTH_CLIENT_ID,
WEBUI_URL, etc.) are left completely untouched -- MCPO_API_KEY migration
is deferred to a follow-up per plan, and this change never reads or
writes that value.
New secrets required in Woodpecker (Settings -> Secrets) before merge:
ai_aws_access_key_id
ai_aws_secret_access_key
ai_litellm_master_key
ai_litellm_salt_key
ai_litellm_db_password
ai_webui_secret_key
ai_open_webui_database_url
ai_oauth_client_secret
Root cause of the "Canary row did not propagate" Phase 3 failure (run
#7): legacy's pg_hba.conf requires scram-sha-256 for any non-local
connection, and every remote `psql -h <patroni-node>` call in this
script had never supplied a password at all. This was masked until the
prior stderr-capture fix (run #7) surfaced the real error:
"fe_sendauth: no password supplied" on every single attempt.
patroni-0/patroni-1 use the SAME PGadmin superuser + SAME
postgresql_password secret as legacy itself (per
postgresql-ha-staging.yaml), so the fix reads that secret once via
`docker exec "$LEGACY_CID" cat /run/secrets/postgresql_password` early
in Phase 3, then passes it to every remote psql call via
`docker exec -e PGPASSWORD=...` (not spliced into the bash -c string,
to avoid quoting hazards).
Found and fixed the identical missing-password pattern in THREE
places, all with the same root cause:
- Phase 3: the canary-propagation SELECT (where it was first caught)
- Phase 5: the post-promotion pg_is_in_recovery() check
- Phase 8: the alias write + both leader/replica visibility checks
Added "2026 run #8" entry to the script's own header FIX LOG. Not yet
re-validated by a run reaching past Phase 3.
See ADR-0001 note, Session Update 11 (to be added).
Phase 3's "Canary row did not propagate to <replica> within 5s" failure
has now recurred twice (2026 run #4, root-caused as the Phase 2 lag-check
bug; and the run immediately after the Ceph IOPS fix, cause unconfirmed)
with genuinely healthy Phase 1/2 beforehand both times. The per-attempt
SELECT against the replica was discarding stderr entirely (2>/dev/null),
so a real connection/auth error and a genuine multi-second replication
delay were indistinguishable in the log — both just showed "row not
found".
Each of the 5 propagation-check attempts now captures stderr to
/tmp/cutover_phase3_attempt_<N>.stderr (mirrors the existing Phase 8
pattern) and echoes result+stderr into the log per-attempt. The final
trigger_rollback() message on failure includes the last non-empty
stderr seen directly in the FAIL line. The initial canary write's
stderr is also no longer discarded. No behavior change to timing/retry
counts — purely additive diagnostics.
See ADR-0001 note, Session Update 10 (to be added).
Persists this run's entire stdout/stderr transcript to BACKUP_DIR
(/volume1/SMB-docker/backup) — the same directory the pg_dumpall
backup lands in. Appends to an already-open CUTOVER_SESSION_LOG if
invoked as a child of cutover.sh (merging into that session's single
transcript); opens its own rollback-standalone-<TS>.log if run
standalone (including a manual run long after the fact, per this
script's own asymmetry warning). This is the single highest-value
place for a durable transcript in the whole suite, since rollback.sh
failing partway is the one scenario RUNBOOK.md flags as requiring
manual intervention. Also logs the transcript path in the grace-window
refusal message so it's not lost even in that failure mode.
Console/SSH output unchanged (tee mirrors to both).
See ADR-0001 note, Session Update 9.
Persists the ENTIRE multi-phase transcript (Pre-Phase-0 through Phase
11, including Phase 0's preflight.sh output and any automatically
triggered rollback.sh output) to a single timestamped
cutover-session-<TS>.log in BACKUP_DIR (/volume1/SMB-docker/backup) —
the same directory the pg_dumpall backup lands in. Exports
CUTOVER_SESSION_LOG so child preflight.sh/rollback.sh invocations
append to the same file instead of opening their own. Adds an EXIT
trap that always announces final exit code + log path, specifically
so the one scenario RUNBOOK.md flags as needing manual intervention
(rollback.sh itself failing partway) is still fully investigable
after the fact even without a live terminal. Console/SSH output is
unchanged (tee mirrors to both).
See ADR-0001 note, Session Update 9.
Persists this run's entire stdout/stderr transcript to BACKUP_DIR
(/volume1/SMB-docker/backup), the SAME location as the pg_dumpall
backup file itself, so a failed run can be investigated later even
without a live terminal attached. Appends to an already-open
CUTOVER_SESSION_LOG if invoked as a child of cutover.sh (one merged
multi-phase transcript per session); opens its own
preflight-standalone-<TS>.log if run directly. Also logs the repo's
git HEAD at cutover/ for traceability, per the ADR-0001 note's
local-checkout-drift lesson (Session Update 8).
See ADR-0001 note, Session Update 9.
Real incident: a run had both patroni-0 and patroni-1 stuck forever on
"waiting for standby_leader to bootstrap", never even attempting to race
for the role. Root cause was leftover etcd/patroni data on disk from a
prior interrupted run (operator stopped it short) — the postgresqlha
stack itself was gone, but etcd-1/2/3-data still had persisted raft state
including a real /service/postgres-ha/initialize key and old replication
slot records. Fresh Patroni nodes booting against that non-fresh etcd
correctly concluded the cluster already existed and deferred forever
waiting for a leader that could never appear, since nobody actually held
the lock. rollback.sh's own data-dir wipe only fires when it detects the
HA stack IS currently present (Case D/E) — if the stack was already gone
by the time cleanup ran, its Case A path never touches the data dirs,
leaving exactly this trap.
Added a "Pre-Phase-0" check that runs before preflight.sh's ~6+ minute
pg_dumpall: detects a still-present postgresqlha stack OR non-empty
etcd-*/patroni-*-data left over from a prior run, and — since this is
destructive and the operator explicitly wants this to be a deliberate
choice, not silent automatic cleanup — prompts interactively before
tearing down/wiping. Non-interactive sessions (no tty) hard-fail with a
clear message rather than guessing; AUTO_CLEANUP=yes in the environment
skips the prompt for deliberate unattended re-runs. Legacy production
data is never touched by any of this.
Root cause of a real run's failure: Phase 2 passed (patroni-0 reached
standby_leader, patroni-1's basebackup completed, role flipped to
"replica"), but Phase 3's canary write then failed to propagate within
5s moments later. patroni_lag() queried the REPLICA's own /patroni
endpoint for lag data — but that field only exists on the LEADER side
(derived from pg_stat_replication); a replica's own /patroni response
never has it. So $LAG_INFO was always empty, and Phase 2's gate
`[ -z "$LAG_INFO" ] || ...` short-circuited permanently true — the lag
check never actually ran. Phase 2 degraded to "did role flip to replica
3x in a row", which can be true before the replica has genuinely caught
up on WAL backlog from its own basebackup.
Fixed by replacing patroni_lag() with cluster_member_lag_state(), which
queries the LEADER's /cluster endpoint (real pg_stat_replication-backed
data, same shape verified in the original dry run) and extracts the
specific replica's state/lag fields from its member object. Phase 2 now
requires literal state=streaming AND (lag=0 or absent), not just "field
was empty because we asked the wrong node." Phase 4's informational lag
log line updated to match.
admin
AVB