Root cause of PR #16's "failure" (pipelines #398–#401)
The ai deploy actually succeeded completely — provision-stack.py's first production run was flawless (all 10 placeholders resolved, complete 21-key ai.env shipped, flowagent Docker secrets verified-unchanged), stack-deploy.sh ai ran clean, and all 6 ai_* services are healthy right now. The red X was a phantom:
secrets/secrets-map.yaml in the changed-files list made folder-detection (cut -d/ -f1, excluding only deploy/) compute ALL_STACKS="ai secrets". The deploy step survived because secrets coincidentally sits in the bootstrap-tier skip list. The verify step had no guard: docker stack ps secrets exits non-zero, and under errexit an assignment from a failing command substitution kills the step instantly — before verify's own WARNING path (designed for exactly this) could run. The log's abrupt end after --- secrets --- is the tell.
This hazard was first flagged July 2026 in PR #3 ("Add secrets folder as an excluded deploy folder"), which was closed unmerged. It waited for the first commit to ever touch secrets/ — PR #16.
Changes (all in .woodpecker/deploy.yml)
grep -v '^deploy$' → grep -vE '^(deploy|secrets)$' in all five folder-detection sites (validate, provision-secrets ×2, deploy, verify)
verify: || true inside the stack-ps command substitution — a genuinely missing stack now produces the designed WARNING instead of aborting the step (comment documents why)
deploy step: secrets/ tooling is now rsynced to the host mirror unconditionally, same treatment as deploy/ (keeps the host checkout complete for emergency manual provisioning)
Header comment: 2026-09-08 incident note added
Everything else byte-identical to main; $${ escape count audited.
#398 failed in deploy on a transient CephFS rsync verification hiccup ("failed verification -- update discarded"); #399/#400 similar; #401 was the clean run that exposed the verify bug. Only the verify bug is being fixed here — the rsync transient is a known CephFS quirk that rsync itself retries.
After merge
This PR touches only .woodpecker/ → no stack deploys, no service restarts. Next pipeline run on any real stack change validates the fix.
## Root cause of PR #16's "failure" (pipelines #398–#401)
**The ai deploy actually succeeded completely** — provision-stack.py's first production run was flawless (all 10 placeholders resolved, complete 21-key `ai.env` shipped, flowagent Docker secrets verified-unchanged), `stack-deploy.sh ai` ran clean, and all 6 `ai_*` services are healthy right now. The red X was a phantom:
`secrets/secrets-map.yaml` in the changed-files list made folder-detection (`cut -d/ -f1`, excluding only `deploy/`) compute `ALL_STACKS="ai secrets"`. The `deploy` step survived because `secrets` coincidentally sits in the bootstrap-tier skip list. The `verify` step had no guard: `docker stack ps secrets` exits non-zero, and under errexit an assignment from a failing command substitution kills the step instantly — before verify's own WARNING path (designed for exactly this) could run. The log's abrupt end after `--- secrets ---` is the tell.
This hazard was first flagged **July 2026 in PR #3** ("Add secrets folder as an excluded deploy folder"), which was closed unmerged. It waited for the first commit to ever touch `secrets/` — PR #16.
## Changes (all in `.woodpecker/deploy.yml`)
1. `grep -v '^deploy$'` → `grep -vE '^(deploy|secrets)$'` in **all five** folder-detection sites (validate, provision-secrets ×2, deploy, verify)
2. verify: `|| true` inside the stack-ps command substitution — a genuinely missing stack now produces the designed WARNING instead of aborting the step (comment documents why)
3. deploy step: `secrets/` tooling is now rsynced to the host mirror unconditionally, same treatment as `deploy/` (keeps the host checkout complete for emergency manual provisioning)
4. Header comment: 2026-09-08 incident note added
Everything else byte-identical to main; `$${` escape count audited.
## Note on #398 vs #401 (same commit, 4 runs)
#398 failed in `deploy` on a transient CephFS rsync verification hiccup ("failed verification -- update discarded"); #399/#400 similar; #401 was the clean run that exposed the verify bug. Only the verify bug is being fixed here — the rsync transient is a known CephFS quirk that rsync itself retries.
## After merge
This PR touches only `.woodpecker/` → no stack deploys, no service restarts. Next pipeline run on any real stack change validates the fix.
Root cause of PR #16's pipeline failure (#398-#401): secrets/ is a
tooling folder (secrets-map.yaml, *.secrets.example), not a stack, but
folder-detection (cut -d/ -f1, excluding only deploy/) computed
ALL_STACKS="ai secrets" the first time a commit touched secrets/. The
deploy step survived only because 'secrets' coincidentally sits in the
bootstrap-tier skip list; verify had no guard — `docker stack ps
secrets` exited non-zero and, under errexit, the OUTPUT=$(...)
assignment killed the step before the designed WARNING path could run.
The ai deploy itself was fully successful (complete 21-key render, all
6 services healthy) — the failure was pure phantom. Hazard first
flagged July 2026 in PR #3 (closed unmerged).
Changes:
- grep -v '^deploy$' -> grep -vE '^(deploy|secrets)$' in ALL five
folder-detection sites (validate, provision-secrets x2, deploy,
verify)
- verify: `|| true` inside the stack-ps command substitution so a
genuinely missing stack warns instead of aborting (comment added)
- deploy step: also rsync secrets/ tooling to the host mirror
unconditionally, same treatment as deploy/
- header: 2026-09-08 incident note
Everything else byte-identical to main. Dollar-escape audit done.
AVB
merged commit 5a5c8e56a9 into main2026-09-07 22:28:36 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Root cause of PR #16's "failure" (pipelines #398–#401)
The ai deploy actually succeeded completely — provision-stack.py's first production run was flawless (all 10 placeholders resolved, complete 21-key
ai.envshipped, flowagent Docker secrets verified-unchanged),stack-deploy.sh airan clean, and all 6ai_*services are healthy right now. The red X was a phantom:secrets/secrets-map.yamlin the changed-files list made folder-detection (cut -d/ -f1, excluding onlydeploy/) computeALL_STACKS="ai secrets". Thedeploystep survived becausesecretscoincidentally sits in the bootstrap-tier skip list. Theverifystep had no guard:docker stack ps secretsexits non-zero, and under errexit an assignment from a failing command substitution kills the step instantly — before verify's own WARNING path (designed for exactly this) could run. The log's abrupt end after--- secrets ---is the tell.This hazard was first flagged July 2026 in PR #3 ("Add secrets folder as an excluded deploy folder"), which was closed unmerged. It waited for the first commit to ever touch
secrets/— PR #16.Changes (all in
.woodpecker/deploy.yml)grep -v '^deploy$'→grep -vE '^(deploy|secrets)$'in all five folder-detection sites (validate, provision-secrets ×2, deploy, verify)|| trueinside the stack-ps command substitution — a genuinely missing stack now produces the designed WARNING instead of aborting the step (comment documents why)secrets/tooling is now rsynced to the host mirror unconditionally, same treatment asdeploy/(keeps the host checkout complete for emergency manual provisioning)Everything else byte-identical to main;
$${escape count audited.Note on #398 vs #401 (same commit, 4 runs)
#398 failed in
deployon a transient CephFS rsync verification hiccup ("failed verification -- update discarded"); #399/#400 similar; #401 was the clean run that exposed the verify bug. Only the verify bug is being fixed here — the rsync transient is a known CephFS quirk that rsync itself retries.After merge
This PR touches only
.woodpecker/→ no stack deploys, no service restarts. Next pipeline run on any real stack change validates the fix.