homelab/compose-files
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
256 Commits 12 Branches 0 Tags
8f860ab14d55d69987c3392d78d2fd4d8380b4fa
Commit Graph
4 Commits
Author SHA1 Message Date
admin e57f4b3592 cutover.sh: extend Phase 2 cascade-replica wait window to 35 min
ci/woodpecker/push/deploy Pipeline was successful
Details
ci/woodpecker/cron/renovate Pipeline was successful
Details 
Phase 2's cascade-replica bootstrap does its own full basebackup FROM the
new standby_leader (an extra hop beyond Phase 1's legacy-direct copy), so
per operator request its window is extended further than Phase 1's —
from 1200s (20 min, matched to observed 8-11 min legacy-direct timing) to
2100s (35 min), giving more margin for the additional hop. Phase 1's
window is intentionally left unchanged at 1200s since it already has
comfortable headroom against the timing we've actually observed for that
specific bootstrap path.
 2026-08-04 00:48:32 -07:00
admin 29cd73eabc cutover.sh: dynamic standby_leader detection + extended bootstrap wait windows
ci/woodpecker/push/deploy Pipeline was successful
Details 
Two real-run bugs found and fixed:

1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The
   bootstrap-race winner is actually nondeterministic (Patroni/etcd lock
   race) — the original prod attempt had patroni-0 win it instead, which
   the dry run never exercised. Fixed by polling BOTH patroni-0 and
   patroni-1 each iteration and capturing whichever wins into
   $LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase
   (2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of
   hardcoded hostnames.

2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter
   than the observed real basebackup duration for a ~42GB cluster
   (8-11 minutes per prior dry-run/live polling). The role only flips to
   standby_leader/replica AFTER the full copy completes, so both timeouts
   could fire — and did — while a legitimate basebackup was still
   in-progress, triggering a false-negative rollback. Both windows
   extended to 1200s (20 min), with per-poll data-dir size logging
   (timeout-guarded du -sh) so progress is observable instead of a silent
   binary wait.
 2026-08-04 00:40:20 -07:00
admin 325daca86b ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks
ci/woodpecker/push/deploy Pipeline was successful
Details
2026-08-02 22:14:17 -07:00
admin 1203630bb5 ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)
ci/woodpecker/push/deploy Pipeline was successful
Details
2026-08-02 17:32:42 -07:00