Commit Graph
3 Commits
Author SHA1 Message Date
admin 29cd73eabc cutover.sh: dynamic standby_leader detection + extended bootstrap wait windows
ci/woodpecker/push/deploy Pipeline was successful
Two real-run bugs found and fixed:

1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The
   bootstrap-race winner is actually nondeterministic (Patroni/etcd lock
   race) — the original prod attempt had patroni-0 win it instead, which
   the dry run never exercised. Fixed by polling BOTH patroni-0 and
   patroni-1 each iteration and capturing whichever wins into
   $LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase
   (2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of
   hardcoded hostnames.

2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter
   than the observed real basebackup duration for a ~42GB cluster
   (8-11 minutes per prior dry-run/live polling). The role only flips to
   standby_leader/replica AFTER the full copy completes, so both timeouts
   could fire — and did — while a legitimate basebackup was still
   in-progress, triggering a false-negative rollback. Both windows
   extended to 1200s (20 min), with per-poll data-dir size logging
   (timeout-guarded du -sh) so progress is observable instead of a silent
   binary wait.
2026-08-04 00:40:20 -07:00
admin 325daca86b ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 22:14:17 -07:00
admin 1203630bb5 ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 17:32:42 -07:00