7 Commits

Include renames

Author	SHA1	Message	Date
AVB	7a62b1b585	Reducing MAX_WAIT back down.	2026-08-05 12:43:35 -07:00
AVB	57354b6613	Increased MAX_WAIT periods for PHASE 1 and PHASE 2	2026-08-05 11:48:23 -07:00
admin	b2b86a668d	cutover.sh: fix Phase 2's lag check — was a silent no-op ... Root cause of a real run's failure: Phase 2 passed (patroni-0 reached standby_leader, patroni-1's basebackup completed, role flipped to "replica"), but Phase 3's canary write then failed to propagate within 5s moments later. patroni_lag() queried the REPLICA's own /patroni endpoint for lag data — but that field only exists on the LEADER side (derived from pg_stat_replication); a replica's own /patroni response never has it. So $LAG_INFO was always empty, and Phase 2's gate `[ -z "$LAG_INFO" ] || ...` short-circuited permanently true — the lag check never actually ran. Phase 2 degraded to "did role flip to replica 3x in a row", which can be true before the replica has genuinely caught up on WAL backlog from its own basebackup. Fixed by replacing patroni_lag() with cluster_member_lag_state(), which queries the LEADER's /cluster endpoint (real pg_stat_replication-backed data, same shape verified in the original dry run) and extracts the specific replica's state/lag fields from its member object. Phase 2 now requires literal state=streaming AND (lag=0 or absent), not just "field was empty because we asked the wrong node." Phase 4's informational lag log line updated to match.	2026-08-05 08:11:15 -07:00
admin	e57f4b3592	cutover.sh: extend Phase 2 cascade-replica wait window to 35 min ... Phase 2's cascade-replica bootstrap does its own full basebackup FROM the new standby_leader (an extra hop beyond Phase 1's legacy-direct copy), so per operator request its window is extended further than Phase 1's — from 1200s (20 min, matched to observed 8-11 min legacy-direct timing) to 2100s (35 min), giving more margin for the additional hop. Phase 1's window is intentionally left unchanged at 1200s since it already has comfortable headroom against the timing we've actually observed for that specific bootstrap path.	2026-08-04 00:48:32 -07:00
admin	29cd73eabc	cutover.sh: dynamic standby_leader detection + extended bootstrap wait windows ... Two real-run bugs found and fixed: 1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The bootstrap-race winner is actually nondeterministic (Patroni/etcd lock race) — the original prod attempt had patroni-0 win it instead, which the dry run never exercised. Fixed by polling BOTH patroni-0 and patroni-1 each iteration and capturing whichever wins into $LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase (2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of hardcoded hostnames. 2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter than the observed real basebackup duration for a ~42GB cluster (8-11 minutes per prior dry-run/live polling). The role only flips to standby_leader/replica AFTER the full copy completes, so both timeouts could fire — and did — while a legitimate basebackup was still in-progress, triggering a false-negative rollback. Both windows extended to 1200s (20 min), with per-poll data-dir size logging (timeout-guarded du -sh) so progress is observable instead of a silent binary wait.	2026-08-04 00:40:20 -07:00
admin	325daca86b	ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks	2026-08-02 22:14:17 -07:00
admin	1203630bb5	ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)	2026-08-02 17:32:42 -07:00