Commit Graph
21 Commits
Author SHA1 Message Date
admin e57f4b3592 cutover.sh: extend Phase 2 cascade-replica wait window to 35 min
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
Phase 2's cascade-replica bootstrap does its own full basebackup FROM the
new standby_leader (an extra hop beyond Phase 1's legacy-direct copy), so
per operator request its window is extended further than Phase 1's —
from 1200s (20 min, matched to observed 8-11 min legacy-direct timing) to
2100s (35 min), giving more margin for the additional hop. Phase 1's
window is intentionally left unchanged at 1200s since it already has
comfortable headroom against the timing we've actually observed for that
specific bootstrap path.
2026-08-04 00:48:32 -07:00
admin 29cd73eabc cutover.sh: dynamic standby_leader detection + extended bootstrap wait windows
ci/woodpecker/push/deploy Pipeline was successful
Two real-run bugs found and fixed:

1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The
   bootstrap-race winner is actually nondeterministic (Patroni/etcd lock
   race) — the original prod attempt had patroni-0 win it instead, which
   the dry run never exercised. Fixed by polling BOTH patroni-0 and
   patroni-1 each iteration and capturing whichever wins into
   $LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase
   (2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of
   hardcoded hostnames.

2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter
   than the observed real basebackup duration for a ~42GB cluster
   (8-11 minutes per prior dry-run/live polling). The role only flips to
   standby_leader/replica AFTER the full copy completes, so both timeouts
   could fire — and did — while a legitimate basebackup was still
   in-progress, triggering a false-negative rollback. Both windows
   extended to 1200s (20 min), with per-poll data-dir size logging
   (timeout-guarded du -sh) so progress is observable instead of a silent
   binary wait.
2026-08-04 00:40:20 -07:00
admin 9c1f31a70e ADR-0001 rollback.sh: accept any 2xx in consumer health checks (204 from Woodpecker was a false-positive failure)
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
2026-08-02 22:15:15 -07:00
admin 325daca86b ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 22:14:17 -07:00
admin 066cbaf97f ADR-0001 Phase 3: add rollback.sh (standalone, idempotent, grace-window guard against post-cutover data loss)
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
2026-08-02 17:38:50 -07:00
admin 1203630bb5 ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 17:32:42 -07:00
admin a4f5d54b82 ADR-0001 Phase 3: add postgresql-ha-final.yaml (Stage 3 - HAProxy alias handoff, no external port/Traefik yet)
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 17:28:26 -07:00
admin 01768d14f5 ADR-0001 Phase 3: add preflight.sh (disk/backup hard gates, catalog+consumer enumeration)
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 16:45:44 -07:00
admin f49b576806 ADR-0001 Phase 3: add cutover RUNBOOK (dependency map, phased procedure, rollback)
ci/woodpecker/push/deploy Pipeline was successful
2026-08-02 16:35:52 -07:00
admin 4dcb3561ed postgresql-ha-staging.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming, validated in pgha-test dry run) - closes pre-cutover write gap
ci/woodpecker/push/deploy Pipeline was successful
2026-07-30 21:59:44 -07:00
admin dc6120ffd4 pgha-dryrun.yaml: trigger secret re-provisioning after regenerating postgresql_replication_password (excludes &<>\" per pystache HTML-escaping bug found this session)
ci/woodpecker/push/deploy Pipeline was successful
2026-07-30 21:44:49 -07:00
admin 8afe0049c4 pgha-dryrun.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming) to close the pre-cutover write gap
ci/woodpecker/push/deploy Pipeline was successful
2026-07-30 21:24:28 -07:00
admin 499402cd0d postgresql-ha-staging.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping (incl. CLONE_PASSWORD), ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
2026-07-30 15:56:43 -07:00
admin b7f0d9dbfb pgha dry-run: fix bug 5 - grant SUPERUSER to placeholder postgres role
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
2026-07-29 09:32:09 -07:00
admin e8e506dc4d add: post_init_wrapper.sh to work around Spilo's hardcoded 'postgres' role name assumption in post_init.sh. Does not fork/modify Spilo's script — creates missing role then execs the original unchanged.
ci/woodpecker/push/deploy Pipeline was successful
2026-07-29 07:21:34 -07:00
admin 7529e6cb36 fix: override bootstrap.post_init via SPILO_CONFIGURATION to create missing 'postgres' role before Spilo's real post_init.sh runs. Spilo hardcodes ALTER VIEW...OWNER TO postgres with no way to parameterize, which fails since our superuser is PGadmin not postgres.
ci/woodpecker/push/deploy Pipeline was successful
2026-07-29 06:35:24 -07:00
admin b86784fe3a fix: legacy container needs pg_hba.conf replication rule for pg_basebackup — added initdb.d hook script. Disposable test only, uses 'trust' since this container is not auth-representative.
ci/woodpecker/push/deploy Pipeline was successful
2026-07-28 21:27:13 -07:00
admin 1dcb5ec467 fix: use ETCD3_HOSTS not ETCD_HOSTS — confirmed against spilo source that etcd/etcd3 are distinct DCS backends (v2 vs v3 API). Our etcd 3.5.9 containers have v2 API disabled, causing 404s with the old var name.
ci/woodpecker/push/deploy Pipeline was successful
2026-07-28 21:10:13 -07:00
admin 83480aa23b fix: escape \$(...) as \$\$(...) in patroni command blocks — Compose interpolation was choking on \$( before the shell ever saw it (invalid interpolation format error)
ci/woodpecker/push/deploy Pipeline was successful
2026-07-28 21:03:56 -07:00
admin 20e1210441 postgresql: add disposable dry-run test stack for cutover validation (pgha-test, own network + data dirs, zero prod impact)
ci/woodpecker/push/deploy Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
2026-07-28 11:12:19 -07:00
admin f4f0749969 postgresql: add cutover staging compose (ADR-0001 Phase 3). Lives in cutover/ subdir — deliberately excluded from stack-deploy.sh folder merge. Deployed only by cutover script as separate postgresqlha stack.
ci/woodpecker/push/deploy Pipeline was successful
2026-07-28 11:07:03 -07:00