verify: fix race condition causing false-alarm "nothing found in stack" #5

Closed
admin wants to merge 1 commits from fix-verify-race-condition into main
Owner

What

Fixes the false-alarm "nothing found in stack: X" output seen in the verify step (e.g. on the vaultwarden deploy on 2026-08-25).

Root cause

docker stack deploy briefly tears down and recreates a service's tasks in Swarm's internal bookkeeping during a normal update. The verify step did a single sleep 5 and then one docker stack ps check — if that landed during the brief recreate window, it printed "nothing found in stack" even though the service was healthy seconds later. Confirmed on the vaultwarden incident: the service was 1/1 running and had a normal UpdatedAt timestamp immediately after.

Fix

Replaces the fixed sleep 5 + single check with a retry loop:

  • Up to 6 attempts, 5s apart (~30s total budget)
  • Prints the running tasks as soon as any are found
  • If still empty after all attempts, prints a WARNING (not a pipeline failure — verify was already informational) with the exact manual command to check: docker stack ps <stack> --no-trunc

Scope

Only the verify step changed. provision-secrets and deploy steps are untouched (this diff sits on top of the already-merged AI secrets migration).

Risk

Low — this step doesn't gate deploys (no exit 1 on failure), only observability/logging. Worst case if something's wrong: same as before, a manual check is needed, just with clearer messaging and one extra ~30s tail on the pipeline.

Verification after merge

Push any trivial change to a stack (or just watch the next real deploy) and confirm the verify step logs show the retry behavior working, and no more spurious "nothing found" on a healthy stack.

## What Fixes the false-alarm "nothing found in stack: X" output seen in the `verify` step (e.g. on the `vaultwarden` deploy on 2026-08-25). ## Root cause `docker stack deploy` briefly tears down and recreates a service's tasks in Swarm's internal bookkeeping during a normal update. The `verify` step did a single `sleep 5` and then one `docker stack ps` check — if that landed during the brief recreate window, it printed "nothing found in stack" even though the service was healthy seconds later. Confirmed on the vaultwarden incident: the service was `1/1` running and had a normal `UpdatedAt` timestamp immediately after. ## Fix Replaces the fixed `sleep 5` + single check with a retry loop: - Up to 6 attempts, 5s apart (~30s total budget) - Prints the running tasks as soon as any are found - If still empty after all attempts, prints a `WARNING` (not a pipeline failure — `verify` was already informational) with the exact manual command to check: `docker stack ps <stack> --no-trunc` ## Scope Only the `verify` step changed. `provision-secrets` and `deploy` steps are untouched (this diff sits on top of the already-merged AI secrets migration). ## Risk Low — this step doesn't gate deploys (no `exit 1` on failure), only observability/logging. Worst case if something's wrong: same as before, a manual check is needed, just with clearer messaging and one extra ~30s tail on the pipeline. ## Verification after merge Push any trivial change to a stack (or just watch the next real deploy) and confirm the `verify` step logs show the retry behavior working, and no more spurious "nothing found" on a healthy stack.
admin added 1 commit 2026-08-25 23:07:11 -07:00
`docker stack deploy` briefly tears down and recreates tasks in Swarm's
internal bookkeeping, so `docker stack ps` can transiently return nothing
right after a deploy even when the service is healthy. A single `sleep 5`
followed by one check produced a false-alarm-looking "nothing found in
stack: vaultwarden" on an otherwise-successful deploy (2026-08-25).

Now retries up to 6 times, 5s apart (~30s total) before printing a WARNING
with a manual-check command. Does not fail the pipeline on its own --
verify was already informational, not a hard gate.

No changes to provision-secrets or deploy steps.
admin closed this pull request 2026-08-25 23:47:35 -07:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.