postgresql-ha-staging.yaml: add primary_slot_name to fix real bootstrap failure

Root cause of a real cutover run's failure: the standby_leader's basebackup
from legacy completed, but its Postgres process then got permanently stuck
in "starting" because legacy had no replication slot reserving WAL — normal
WAL recycling (checkpoint_timeout=300s) deleted the segment needed to
resume streaming during the 8-11 min basebackup window. This also explained
why the cascade replica's own basebackup (which targets the standby_leader)
failed with "database system is starting up" — one root cause, not two.

Fix: created a physical replication slot (standby_leader_slot) on live
production legacy, and added primary_slot_name: standby_leader_slot under
bootstrap.dcs.standby_cluster in both patroni-0 and patroni-1's
SPILO_CONFIGURATION so Patroni pins the slot automatically. Also bumped
wal_keep_size to 4GB on legacy as defense-in-depth.

Full incident detail documented in this file's header for future reference.

postgresql/cutover/postgresql-ha-staging.yaml

@@ -25,8 +25,8 @@
 #   - Superuser is PGadmin (matches the cloned data's existing superuser,
 #     same password secret), NOT Spilo's default "postgres".
 #
-# ─── REVISION (2026-08-01): switched from CLONE_WITH_BASEBACKUP to a
-# Patroni "standby cluster" (continuous streaming) ───
+# ── REVISION (2026-08-01): switched from CLONE_WITH_BASEBACKUP to a
+# Patroni "standby cluster" (continuous streaming) ──
 #
 # WHY: CLONE_WITH_BASEBACKUP is a ONE-SHOT snapshot — once pg_basebackup
 # completes, the new cluster has ZERO further connection to the live
@@ -43,8 +43,7 @@
 # cascade replica in <3s, and a full promote-with-zero-data-loss cycle —
 # see "ADR-0001 Dry-Run Debugging Log" note for the complete play-by-play.
 #
-# ⚠️ NEW PRODUCTION PREREQUISITE THIS INTRODUCES — NOT YET DONE, must
-# happen before this file can actually be deployed for real: the dry run's
+# ⚠️ NEW PRODUCTION PREREQUISITE THIS INTRODUCES — the dry run's
 # "legacy" stand-in creates a genuine replication role named "standby"
 # (matching PGUSER_STANDBY below) via a fresh-initdb hook script, since
 # standby_cluster streaming authenticates using the cluster's real
@@ -60,8 +59,69 @@
 # This is NOT part of this compose file (can't be — it's a one-time SQL
 # statement against already-running production, not something a compose
 # file can express) but MUST be tracked and done as an explicit manual
-# step in the real Phase 3 procedure. See ADR-0001 Dry-Run Debugging Log
-# note, "Not yet done" section, for status.
+# step in the real Phase 3 procedure. STATUS: DONE — see ADR-0001 note,
+# "Production prerequisite" section, for full verification detail
+# (CREATE ROLE + pg_hba.conf line, both proven end-to-end).
+#
+# ── REVISION 2 (2026-08-04): added primary_slot_name to standby_cluster —
+# fixes a real bootstrap failure hit on a live production run ──
+#
+# WHAT HAPPENED: on a real cutover attempt, the standby_leader's OWN
+# basebackup from legacy completed successfully, but the resulting
+# Postgres process then got stuck in "starting"/"rejecting connections"
+# PERMANENTLY (confirmed via the container's own pg_log/*.csv, since
+# docker service logs stops showing Postgres's own LOG lines once the
+# logging collector activates — check
+# <data-dir>/pgroot/pg_log/postgresql-N.csv directly, NOT `docker service
+# logs`, if this needs debugging again). The actual error, repeating
+# every ~5s indefinitely:
+#   LOG:  started streaming WAL from primary at <LSN> on timeline 1
+#   FATAL: could not receive data from WAL stream: ERROR: requested WAL
+#          segment <segment> has already been removed
+#   LOG:  waiting for WAL to become available at <LSN>
+# ROOT CAUSE: legacy had NO replication slot reserving WAL for this
+# connection (confirmed: `select * from pg_replication_slots` = 0 rows,
+# wal_keep_size = 0, max_slot_wal_keep_size = -1/irrelevant with no slot).
+# basebackup_fast_xlog copies data files, then Postgres must replay WAL
+# FORWARD from the backup's start LSN via a normal streaming connection to
+# catch up. With no slot, legacy's normal WAL recycling
+# (checkpoint_timeout=300s) deleted the exact segment needed to resume
+# during the 8-11 minute basebackup window — once gone, gone forever, and
+# the standby_leader can never leave "starting". This ALSO explains why
+# the OTHER node's (the cascade replica's) basebackup then failed with
+# "the database system is starting up" — it targets the standby_leader,
+# which never actually finished starting. Not two bugs — one root cause
+# propagating downstream exactly as it would look if it were two.
+# (Secondary, non-blocking finding from the same run: ~2.2GB of core dump
+# files appeared in the standby_leader's data dir root at the moment of
+# basebackup completion — not yet root-caused, possibly related to
+# bg_mon's repeated port-8080-bind failures inside the container, but NOT
+# confirmed as the same issue and NOT blocking this fix.)
+# THE FIX (this revision): a physical replication slot named
+# "standby_leader_slot" was created on live production legacy:
+#   SELECT pg_create_physical_replication_slot('standby_leader_slot');
+# and `primary_slot_name: standby_leader_slot` was added under
+# bootstrap.dcs.standby_cluster in BOTH patroni-0 and patroni-1's
+# SPILO_CONFIGURATION below (Patroni's standby_cluster feature reads this
+# key and uses it to pin the slot on the external primary automatically —
+# no additional Patroni-side config needed). As defense-in-depth,
+# `wal_keep_size` was ALSO bumped from 0 to 4096MB (4GB) directly on live
+# legacy (`ALTER SYSTEND SET wal_keep_size = '4096MB'; SELECT
+# pg_reload_conf();` — NOTE: ALTER SYSTEM cannot share a transaction with
+# other statements, run as two separate -c calls) — belt-and-suspenders
+# in case the slot itself is ever inadvertently dropped mid-cutover.
+# WHY THIS DIDN'T SURFACE EARLIER: the pgha-test dry run used a tiny
+# disposable database with a near-instant basebackup, leaving no real
+# window for WAL to churn before streaming resumed. It's plausible an even
+# earlier real attempt got lucky on timing. With a real ~42GB dataset and
+# multi-minute basebackups, the gap is real and reproducible.
+# STATUS: slot created and verified on live production
+# (`select slot_name, slot_type, active from pg_replication_slots` shows
+# standby_leader_slot/physical/f — inactive is expected/correct until a
+# standby_cluster actually connects to it). wal_keep_size verified at 4GB.
+# This compose file updated to reference the slot. NOT YET RE-RUN against
+# production as of this revision — see ADR-0001 note's "Not yet done"
+# section for current status.
 #
 # NOTE: command: blocks use $$(...) not $(...) — Compose's own variable
 # interpolation parses $( as an attempted ${VAR} reference and fails with
@@ -212,6 +272,7 @@ services:
             standby_cluster:
               host: postgresql
               port: 5432
+              primary_slot_name: standby_leader_slot
               create_replica_methods:
               - basebackup_fast_xlog
     secrets:
@@ -261,6 +322,7 @@ services:
             standby_cluster:
               host: postgresql
               port: 5432
+              primary_slot_name: standby_leader_slot
               create_replica_methods:
               - basebackup_fast_xlog
     secrets: