Comment-only change. Forces a real deploy of the ai stack now that
PR #8 (envparse.py $$ escaping) and PR #9 (deploy/ folder exclusion)
are both merged, so litellm picks up the correctly-escaped
LITELLM_MASTER_KEY/LITELLM_SALT_KEY instead of the truncated values
currently running (truncated at the first literal '$' due to the
envsubst+Compose double-interpolation bug fixed in #8).
grep -E '^[^/.][^/]*/' matches ANY non-dot top-level folder in the
changed-files list, including deploy/ -- the shared tooling folder
synced by every deploy, not a stack. A PR touching only
deploy/envparse.py caused stack-deploy.sh to be invoked with "deploy"
as a stack name, which correctly errored ("No main compose file... in
.../deploy") since deploy/ has no deploy.yaml.
No live service was affected (the error occurs before any redeploy
attempt), but it produced a confusing FAIL on an otherwise-correct
change (PR #8) and could mask a real failure in the noise.
Adds `| grep -v '^deploy$'` after the folder-name extraction in all 5
places this pattern appears (validate, provision-secrets x2, deploy,
verify). deploy/ is already unconditionally rsynced at the top of the
deploy step regardless of which stacks changed, so excluding it from
the stack list is safe -- it will still be synced, just never treated
as a deployable stack.
Root cause of the LITELLM_MASTER_KEY/LITELLM_SALT_KEY truncation
incident (2026-08-26): stack-deploy.sh's single-file deploy path is
`envsubst "$VARS" < stack.yaml | docker stack deploy -c - stack`.
envsubst embeds the raw secret value into the compose YAML text. If
that value contains a literal '$' followed by word chars, the
resulting YAML now contains what looks like a second variable
reference. `docker stack deploy -c -` runs Compose's own interpolation
pass on that text before creating the service, finds no such env var,
and silently substitutes empty string -- truncating the secret in the
running container with no error.
Confirmed: an 87-char LITELLM_MASTER_KEY arrived in the ai_litellm
container as 73 chars, silently, on a real deploy.
This is not specific to ai -- it affects every Pattern B stack (host
.env + envsubst, not native Docker secrets): maintenance, media,
unifi, guacamole, security, auth, traefik, meshcentral, ddm. Any of
them could have a '$'-containing value truncating right now without
detection, since the failure produces no warning.
Fix: escape every literal '$' as '$$' in export/export_merged (which
feed the `eval` that sets envsubst's actual source values), before
envsubst ever sees them. envsubst does not interpret '$' in replacement
text, so the doubled dollar survives envsubst intact; Compose's own
interpolation pass then consumes exactly one level of escaping,
landing on the correct single '$' with no leftover false variable
reference. vars/vars_merged (envsubst's allowlist string, unrelated to
values) are untouched.
NOT deployed/merged yet -- pending review. The currently-running
ai_litellm service still has the truncated keys and needs a fresh
`stack-deploy.sh ai` run after this merges to pick up the corrected
values.
No functional change -- this comment-only edit exists to trigger a real
deploy of the ai stack so provision-secrets' ai) case, and the AI_
var-name fix from PR #7, get exercised end-to-end for the first time.
Documents that MCPO_API_KEY is intentionally still manual/unmigrated.
ai.yaml's litellm service (as of commit 37ed671a, "Change AWS keys to
use Woodpecker Secrets") references ${AI_AWS_ACCESS_KEY_ID} /
${AI_AWS_SECRET_ACCESS_KEY} and renders them into the container as
plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. The ai) provisioning
case added in the earlier secrets-migration PR wrote the plain
(unprefixed) names into ai.env instead, which would leave
${AI_AWS_ACCESS_KEY_ID} unresolved at compose-render time (renders
empty) -- silently breaking Bedrock auth in litellm on the next ai
stack deploy.
Fixed both the grep -vE exclusion pattern and the two printf lines to
use the AI_-prefixed names. All other migrated vars in ai.yaml use
plain names and are unaffected.
No other changes in this file.
The AI secrets migration PR (#4) was a full-file rewrite of deploy.yml.
That rewrite mechanically dropped one $ from EVERY $${VAR} occurrence in
the file, not just the new AI additions -- silently reverting all
pre-existing secret references (SWARM_MANAGER_IP, IMMICH_*, GIT_*,
POSTGRESQL_*, VAULTWARDEN_*, ENTERTAINMENT_*, etc.) to single-dollar
form. Per this file's own header comment, Woodpecker blanks single-dollar
braced refs at compile time since secrets aren't in that variable map --
this is the exact "SWARM_MANAGER_IP secret is empty" failure mode
documented above, and it fired immediately on the first push after #4
merged.
Impact: provision-secrets/deploy/verify all failed at their first
if-empty guard and exited before any ssh/scp/rsync ran. No live secret,
service, or deployed stack was touched -- this was a CI-only outage.
Fix: restored $${VAR} for every secret-backed reference throughout the
file. CI_PIPELINE_FILES / CI_COMMIT_MESSAGE stay single-dollar (correct
-- those are Woodpecker compile-time metadata, not secrets). The \$FILE
/ \$TMP backslash-escaping inside the ai) case's remote SSH command is
unrelated and was already correct (it protects those local-to-remote
vars from expanding before the SSH payload is sent).
This is a straight revert-of-the-regression -- no new secrets, no logic
changes beyond restoring the escaping.
No stack files changed -- this commit exists only to trigger a fresh
Woodpecker pipeline run against current main + current secrets, since
"Restart" on the prior failed run was replaying a stale snapshot from
before the ai_* secrets existed.
Adds 8 new from_secret-backed env vars to provision-secrets and rewrites
the `ai)` case to do a targeted update of only those 8 keys in the
remote ai/ai.env via grep -v + printf (no sed, safe for values containing
/, $, &, etc). All other lines in ai.env (MCPO_API_KEY, OAUTH_CLIENT_ID,
WEBUI_URL, etc.) are left completely untouched -- MCPO_API_KEY migration
is deferred to a follow-up per plan, and this change never reads or
writes that value.
New secrets required in Woodpecker (Settings -> Secrets) before merge:
ai_aws_access_key_id
ai_aws_secret_access_key
ai_litellm_master_key
ai_litellm_salt_key
ai_litellm_db_password
ai_webui_secret_key
ai_open_webui_database_url
ai_oauth_client_secret
Root cause of the "Canary row did not propagate" Phase 3 failure (run
#7): legacy's pg_hba.conf requires scram-sha-256 for any non-local
connection, and every remote `psql -h <patroni-node>` call in this
script had never supplied a password at all. This was masked until the
prior stderr-capture fix (run #7) surfaced the real error:
"fe_sendauth: no password supplied" on every single attempt.
patroni-0/patroni-1 use the SAME PGadmin superuser + SAME
postgresql_password secret as legacy itself (per
postgresql-ha-staging.yaml), so the fix reads that secret once via
`docker exec "$LEGACY_CID" cat /run/secrets/postgresql_password` early
in Phase 3, then passes it to every remote psql call via
`docker exec -e PGPASSWORD=...` (not spliced into the bash -c string,
to avoid quoting hazards).
Found and fixed the identical missing-password pattern in THREE
places, all with the same root cause:
- Phase 3: the canary-propagation SELECT (where it was first caught)
- Phase 5: the post-promotion pg_is_in_recovery() check
- Phase 8: the alias write + both leader/replica visibility checks
Added "2026 run #8" entry to the script's own header FIX LOG. Not yet
re-validated by a run reaching past Phase 3.
See ADR-0001 note, Session Update 11 (to be added).
Phase 3's "Canary row did not propagate to <replica> within 5s" failure
has now recurred twice (2026 run #4, root-caused as the Phase 2 lag-check
bug; and the run immediately after the Ceph IOPS fix, cause unconfirmed)
with genuinely healthy Phase 1/2 beforehand both times. The per-attempt
SELECT against the replica was discarding stderr entirely (2>/dev/null),
so a real connection/auth error and a genuine multi-second replication
delay were indistinguishable in the log — both just showed "row not
found".
Each of the 5 propagation-check attempts now captures stderr to
/tmp/cutover_phase3_attempt_<N>.stderr (mirrors the existing Phase 8
pattern) and echoes result+stderr into the log per-attempt. The final
trigger_rollback() message on failure includes the last non-empty
stderr seen directly in the FAIL line. The initial canary write's
stderr is also no longer discarded. No behavior change to timing/retry
counts — purely additive diagnostics.
See ADR-0001 note, Session Update 10 (to be added).
Persists this run's entire stdout/stderr transcript to BACKUP_DIR
(/volume1/SMB-docker/backup) — the same directory the pg_dumpall
backup lands in. Appends to an already-open CUTOVER_SESSION_LOG if
invoked as a child of cutover.sh (merging into that session's single
transcript); opens its own rollback-standalone-<TS>.log if run
standalone (including a manual run long after the fact, per this
script's own asymmetry warning). This is the single highest-value
place for a durable transcript in the whole suite, since rollback.sh
failing partway is the one scenario RUNBOOK.md flags as requiring
manual intervention. Also logs the transcript path in the grace-window
refusal message so it's not lost even in that failure mode.
Console/SSH output unchanged (tee mirrors to both).
See ADR-0001 note, Session Update 9.
Persists the ENTIRE multi-phase transcript (Pre-Phase-0 through Phase
11, including Phase 0's preflight.sh output and any automatically
triggered rollback.sh output) to a single timestamped
cutover-session-<TS>.log in BACKUP_DIR (/volume1/SMB-docker/backup) —
the same directory the pg_dumpall backup lands in. Exports
CUTOVER_SESSION_LOG so child preflight.sh/rollback.sh invocations
append to the same file instead of opening their own. Adds an EXIT
trap that always announces final exit code + log path, specifically
so the one scenario RUNBOOK.md flags as needing manual intervention
(rollback.sh itself failing partway) is still fully investigable
after the fact even without a live terminal. Console/SSH output is
unchanged (tee mirrors to both).
See ADR-0001 note, Session Update 9.
Persists this run's entire stdout/stderr transcript to BACKUP_DIR
(/volume1/SMB-docker/backup), the SAME location as the pg_dumpall
backup file itself, so a failed run can be investigated later even
without a live terminal attached. Appends to an already-open
CUTOVER_SESSION_LOG if invoked as a child of cutover.sh (one merged
multi-phase transcript per session); opens its own
preflight-standalone-<TS>.log if run directly. Also logs the repo's
git HEAD at cutover/ for traceability, per the ADR-0001 note's
local-checkout-drift lesson (Session Update 8).
See ADR-0001 note, Session Update 9.
Real incident: a run had both patroni-0 and patroni-1 stuck forever on
"waiting for standby_leader to bootstrap", never even attempting to race
for the role. Root cause was leftover etcd/patroni data on disk from a
prior interrupted run (operator stopped it short) — the postgresqlha
stack itself was gone, but etcd-1/2/3-data still had persisted raft state
including a real /service/postgres-ha/initialize key and old replication
slot records. Fresh Patroni nodes booting against that non-fresh etcd
correctly concluded the cluster already existed and deferred forever
waiting for a leader that could never appear, since nobody actually held
the lock. rollback.sh's own data-dir wipe only fires when it detects the
HA stack IS currently present (Case D/E) — if the stack was already gone
by the time cleanup ran, its Case A path never touches the data dirs,
leaving exactly this trap.
Added a "Pre-Phase-0" check that runs before preflight.sh's ~6+ minute
pg_dumpall: detects a still-present postgresqlha stack OR non-empty
etcd-*/patroni-*-data left over from a prior run, and — since this is
destructive and the operator explicitly wants this to be a deliberate
choice, not silent automatic cleanup — prompts interactively before
tearing down/wiping. Non-interactive sessions (no tty) hard-fail with a
clear message rather than guessing; AUTO_CLEANUP=yes in the environment
skips the prompt for deliberate unattended re-runs. Legacy production
data is never touched by any of this.
Root cause of a real run's failure: Phase 2 passed (patroni-0 reached
standby_leader, patroni-1's basebackup completed, role flipped to
"replica"), but Phase 3's canary write then failed to propagate within
5s moments later. patroni_lag() queried the REPLICA's own /patroni
endpoint for lag data — but that field only exists on the LEADER side
(derived from pg_stat_replication); a replica's own /patroni response
never has it. So $LAG_INFO was always empty, and Phase 2's gate
`[ -z "$LAG_INFO" ] || ...` short-circuited permanently true — the lag
check never actually ran. Phase 2 degraded to "did role flip to replica
3x in a row", which can be true before the replica has genuinely caught
up on WAL backlog from its own basebackup.
Fixed by replacing patroni_lag() with cluster_member_lag_state(), which
queries the LEADER's /cluster endpoint (real pg_stat_replication-backed
data, same shape verified in the original dry run) and extracts the
specific replica's state/lag fields from its member object. Phase 2 now
requires literal state=streaming AND (lag=0 or absent), not just "field
was empty because we asked the wrong node." Phase 4's informational lag
log line updated to match.
Mirrors the fix in postgresql-ha-staging.yaml: added
primary_slot_name: standby_leader_slot to both patroni-0 and patroni-1's
standby_cluster blocks so this file's specs stay byte-identical to
staging's, per this file's own "do not recreate the live promoted
primary/replica" design requirement. See postgresql-ha-staging.yaml
header and the ADR-0001 note for full incident detail.
Root cause of a real cutover run's failure: the standby_leader's basebackup
from legacy completed, but its Postgres process then got permanently stuck
in "starting" because legacy had no replication slot reserving WAL — normal
WAL recycling (checkpoint_timeout=300s) deleted the segment needed to
resume streaming during the 8-11 min basebackup window. This also explained
why the cascade replica's own basebackup (which targets the standby_leader)
failed with "database system is starting up" — one root cause, not two.
Fix: created a physical replication slot (standby_leader_slot) on live
production legacy, and added primary_slot_name: standby_leader_slot under
bootstrap.dcs.standby_cluster in both patroni-0 and patroni-1's
SPILO_CONFIGURATION so Patroni pins the slot automatically. Also bumped
wal_keep_size to 4GB on legacy as defense-in-depth.
Full incident detail documented in this file's header for future reference.
Phase 2's cascade-replica bootstrap does its own full basebackup FROM the
new standby_leader (an extra hop beyond Phase 1's legacy-direct copy), so
per operator request its window is extended further than Phase 1's —
from 1200s (20 min, matched to observed 8-11 min legacy-direct timing) to
2100s (35 min), giving more margin for the additional hop. Phase 1's
window is intentionally left unchanged at 1200s since it already has
comfortable headroom against the timing we've actually observed for that
specific bootstrap path.
Two real-run bugs found and fixed:
1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The
bootstrap-race winner is actually nondeterministic (Patroni/etcd lock
race) — the original prod attempt had patroni-0 win it instead, which
the dry run never exercised. Fixed by polling BOTH patroni-0 and
patroni-1 each iteration and capturing whichever wins into
$LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase
(2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of
hardcoded hostnames.
2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter
than the observed real basebackup duration for a ~42GB cluster
(8-11 minutes per prior dry-run/live polling). The role only flips to
standby_leader/replica AFTER the full copy completes, so both timeouts
could fire — and did — while a legitimate basebackup was still
in-progress, triggering a false-negative rollback. Both windows
extended to 1200s (20 min), with per-poll data-dir size logging
(timeout-guarded du -sh) so progress is observable instead of a silent
binary wait.
AVB
admin