diff --git a/postgresql/cutover/postgresql-ha-staging.yaml b/postgresql/cutover/postgresql-ha-staging.yaml index 5a40faa..263796c 100644 --- a/postgresql/cutover/postgresql-ha-staging.yaml +++ b/postgresql/cutover/postgresql-ha-staging.yaml @@ -25,8 +25,8 @@ # - Superuser is PGadmin (matches the cloned data's existing superuser, # same password secret), NOT Spilo's default "postgres". # -# ─── REVISION (2026-08-01): switched from CLONE_WITH_BASEBACKUP to a -# Patroni "standby cluster" (continuous streaming) ─── +# ── REVISION (2026-08-01): switched from CLONE_WITH_BASEBACKUP to a +# Patroni "standby cluster" (continuous streaming) ── # # WHY: CLONE_WITH_BASEBACKUP is a ONE-SHOT snapshot — once pg_basebackup # completes, the new cluster has ZERO further connection to the live @@ -43,8 +43,7 @@ # cascade replica in <3s, and a full promote-with-zero-data-loss cycle — # see "ADR-0001 Dry-Run Debugging Log" note for the complete play-by-play. # -# ⚠️ NEW PRODUCTION PREREQUISITE THIS INTRODUCES — NOT YET DONE, must -# happen before this file can actually be deployed for real: the dry run's +# ⚠️ NEW PRODUCTION PREREQUISITE THIS INTRODUCES — the dry run's # "legacy" stand-in creates a genuine replication role named "standby" # (matching PGUSER_STANDBY below) via a fresh-initdb hook script, since # standby_cluster streaming authenticates using the cluster's real @@ -60,8 +59,69 @@ # This is NOT part of this compose file (can't be — it's a one-time SQL # statement against already-running production, not something a compose # file can express) but MUST be tracked and done as an explicit manual -# step in the real Phase 3 procedure. See ADR-0001 Dry-Run Debugging Log -# note, "Not yet done" section, for status. +# step in the real Phase 3 procedure. STATUS: DONE — see ADR-0001 note, +# "Production prerequisite" section, for full verification detail +# (CREATE ROLE + pg_hba.conf line, both proven end-to-end). +# +# ── REVISION 2 (2026-08-04): added primary_slot_name to standby_cluster — +# fixes a real bootstrap failure hit on a live production run ── +# +# WHAT HAPPENED: on a real cutover attempt, the standby_leader's OWN +# basebackup from legacy completed successfully, but the resulting +# Postgres process then got stuck in "starting"/"rejecting connections" +# PERMANENTLY (confirmed via the container's own pg_log/*.csv, since +# docker service logs stops showing Postgres's own LOG lines once the +# logging collector activates — check +# /pgroot/pg_log/postgresql-N.csv directly, NOT `docker service +# logs`, if this needs debugging again). The actual error, repeating +# every ~5s indefinitely: +# LOG: started streaming WAL from primary at on timeline 1 +# FATAL: could not receive data from WAL stream: ERROR: requested WAL +# segment has already been removed +# LOG: waiting for WAL to become available at +# ROOT CAUSE: legacy had NO replication slot reserving WAL for this +# connection (confirmed: `select * from pg_replication_slots` = 0 rows, +# wal_keep_size = 0, max_slot_wal_keep_size = -1/irrelevant with no slot). +# basebackup_fast_xlog copies data files, then Postgres must replay WAL +# FORWARD from the backup's start LSN via a normal streaming connection to +# catch up. With no slot, legacy's normal WAL recycling +# (checkpoint_timeout=300s) deleted the exact segment needed to resume +# during the 8-11 minute basebackup window — once gone, gone forever, and +# the standby_leader can never leave "starting". This ALSO explains why +# the OTHER node's (the cascade replica's) basebackup then failed with +# "the database system is starting up" — it targets the standby_leader, +# which never actually finished starting. Not two bugs — one root cause +# propagating downstream exactly as it would look if it were two. +# (Secondary, non-blocking finding from the same run: ~2.2GB of core dump +# files appeared in the standby_leader's data dir root at the moment of +# basebackup completion — not yet root-caused, possibly related to +# bg_mon's repeated port-8080-bind failures inside the container, but NOT +# confirmed as the same issue and NOT blocking this fix.) +# THE FIX (this revision): a physical replication slot named +# "standby_leader_slot" was created on live production legacy: +# SELECT pg_create_physical_replication_slot('standby_leader_slot'); +# and `primary_slot_name: standby_leader_slot` was added under +# bootstrap.dcs.standby_cluster in BOTH patroni-0 and patroni-1's +# SPILO_CONFIGURATION below (Patroni's standby_cluster feature reads this +# key and uses it to pin the slot on the external primary automatically — +# no additional Patroni-side config needed). As defense-in-depth, +# `wal_keep_size` was ALSO bumped from 0 to 4096MB (4GB) directly on live +# legacy (`ALTER SYSTEND SET wal_keep_size = '4096MB'; SELECT +# pg_reload_conf();` — NOTE: ALTER SYSTEM cannot share a transaction with +# other statements, run as two separate -c calls) — belt-and-suspenders +# in case the slot itself is ever inadvertently dropped mid-cutover. +# WHY THIS DIDN'T SURFACE EARLIER: the pgha-test dry run used a tiny +# disposable database with a near-instant basebackup, leaving no real +# window for WAL to churn before streaming resumed. It's plausible an even +# earlier real attempt got lucky on timing. With a real ~42GB dataset and +# multi-minute basebackups, the gap is real and reproducible. +# STATUS: slot created and verified on live production +# (`select slot_name, slot_type, active from pg_replication_slots` shows +# standby_leader_slot/physical/f — inactive is expected/correct until a +# standby_cluster actually connects to it). wal_keep_size verified at 4GB. +# This compose file updated to reference the slot. NOT YET RE-RUN against +# production as of this revision — see ADR-0001 note's "Not yet done" +# section for current status. # # NOTE: command: blocks use $$(...) not $(...) — Compose's own variable # interpolation parses $( as an attempted ${VAR} reference and fails with @@ -212,6 +272,7 @@ services: standby_cluster: host: postgresql port: 5432 + primary_slot_name: standby_leader_slot create_replica_methods: - basebackup_fast_xlog secrets: @@ -261,6 +322,7 @@ services: standby_cluster: host: postgresql port: 5432 + primary_slot_name: standby_leader_slot create_replica_methods: - basebackup_fast_xlog secrets: