303 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
admin	29cd73eabc	cutover.sh: dynamic standby_leader detection + extended bootstrap wait windows ... Two real-run bugs found and fixed: 1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The bootstrap-race winner is actually nondeterministic (Patroni/etcd lock race) — the original prod attempt had patroni-0 win it instead, which the dry run never exercised. Fixed by polling BOTH patroni-0 and patroni-1 each iteration and capturing whichever wins into $LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase (2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of hardcoded hostnames. 2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter than the observed real basebackup duration for a ~42GB cluster (8-11 minutes per prior dry-run/live polling). The role only flips to standby_leader/replica AFTER the full copy completes, so both timeouts could fire — and did — while a legitimate basebackup was still in-progress, triggering a false-negative rollback. Both windows extended to 1200s (20 min), with per-poll data-dir size logging (timeout-guarded du -sh) so progress is observable instead of a silent binary wait.	2026-08-04 00:40:20 -07:00
admin	9c1f31a70e	ADR-0001 rollback.sh: accept any 2xx in consumer health checks (204 from Woodpecker was a false-positive failure)	2026-08-02 22:15:15 -07:00
admin	325daca86b	ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks	2026-08-02 22:14:17 -07:00
admin	066cbaf97f	ADR-0001 Phase 3: add rollback.sh (standalone, idempotent, grace-window guard against post-cutover data loss)	2026-08-02 17:38:50 -07:00
admin	1203630bb5	ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)	2026-08-02 17:32:42 -07:00
admin	a4f5d54b82	ADR-0001 Phase 3: add postgresql-ha-final.yaml (Stage 3 - HAProxy alias handoff, no external port/Traefik yet)	2026-08-02 17:28:26 -07:00
admin	01768d14f5	ADR-0001 Phase 3: add preflight.sh (disk/backup hard gates, catalog+consumer enumeration)	2026-08-02 16:45:44 -07:00
admin	f49b576806	ADR-0001 Phase 3: add cutover RUNBOOK (dependency map, phased procedure, rollback)	2026-08-02 16:35:52 -07:00
admin	8d30f19c8c	postgresql.yaml: update stale comment referencing old CLONE_* rationale to reference standby_cluster design (no functional change - this file never used CLONE_* itself)	2026-07-30 22:03:44 -07:00
admin	4dcb3561ed	postgresql-ha-staging.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming, validated in pgha-test dry run) - closes pre-cutover write gap	2026-07-30 21:59:44 -07:00
admin	dc6120ffd4	pgha-dryrun.yaml: trigger secret re-provisioning after regenerating postgresql_replication_password (excludes &<>\" per pystache HTML-escaping bug found this session)	2026-07-30 21:44:49 -07:00
admin	8afe0049c4	pgha-dryrun.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming) to close the pre-cutover write gap	2026-07-30 21:24:28 -07:00
admin	499402cd0d	postgresql-ha-staging.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping (incl. CLONE_PASSWORD), ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix	2026-07-30 15:56:43 -07:00
admin	7e8e3388f4	postgresql.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping, ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix	2026-07-30 15:55:26 -07:00
admin	b7f0d9dbfb	pgha dry-run: fix bug 5 - grant SUPERUSER to placeholder postgres role	2026-07-29 09:32:09 -07:00
admin	e8e506dc4d	add: post_init_wrapper.sh to work around Spilo's hardcoded 'postgres' role name assumption in post_init.sh. Does not fork/modify Spilo's script — creates missing role then execs the original unchanged.	2026-07-29 07:21:34 -07:00
admin	7529e6cb36	fix: override bootstrap.post_init via SPILO_CONFIGURATION to create missing 'postgres' role before Spilo's real post_init.sh runs. Spilo hardcodes ALTER VIEW...OWNER TO postgres with no way to parameterize, which fails since our superuser is PGadmin not postgres.	2026-07-29 06:35:24 -07:00
admin	b86784fe3a	fix: legacy container needs pg_hba.conf replication rule for pg_basebackup — added initdb.d hook script. Disposable test only, uses 'trust' since this container is not auth-representative.	2026-07-28 21:27:13 -07:00
admin	1dcb5ec467	fix: use ETCD3_HOSTS not ETCD_HOSTS — confirmed against spilo source that etcd/etcd3 are distinct DCS backends (v2 vs v3 API). Our etcd 3.5.9 containers have v2 API disabled, causing 404s with the old var name.	2026-07-28 21:10:13 -07:00
admin	83480aa23b	fix: escape \$(...) as \$\$(...) in patroni command blocks — Compose interpolation was choking on \$( before the shell ever saw it (invalid interpolation format error)	2026-07-28 21:03:56 -07:00
admin	20e1210441	postgresql: add disposable dry-run test stack for cutover validation (pgha-test, own network + data dirs, zero prod impact)	2026-07-28 11:12:19 -07:00
admin	f4f0749969	postgresql: add cutover staging compose (ADR-0001 Phase 3). Lives in cutover/ subdir — deliberately excluded from stack-deploy.sh folder merge. Deployed only by cutover script as separate postgresqlha stack.	2026-07-28 11:07:03 -07:00
admin	87ae54fb3f	postgresql: add HAProxy config for Patroni-aware TCP routing (ADR-0001 Phase 2)	2026-07-28 08:53:55 -07:00
admin	c8d45e9e61	postgresql: add folder-based Patroni+etcd+HAProxy HA stack draft (ADR-0001 Phase 2). NOT deployed — bootstrap tier, manual deploy only. Coexists with flat postgresql.yaml until cutover.	2026-07-28 08:53:44 -07:00
AVB	129badada4	Add secrets (and add postgresql) back so they are seen as utility folders and do deploy	2026-07-28 08:36:56 -07:00
AVB	e1f569b3ec	Add secrets folder to utility folders list so it isn't auto-deployed as a stack	2026-07-28 08:25:35 -07:00
AVB	b5767ec7d9	revert 8719c398fd ... revert Update .woodpecker/deploy.yml	2026-07-28 08:24:29 -07:00
AVB	8719c398fd	Update .woodpecker/deploy.yml	2026-07-28 08:22:57 -07:00
admin	9a5b028e07	postgresql: document replication + patroni secrets (ADR-0001 Phase 1)	2026-07-28 07:20:52 -07:00
admin	0f26d31a02	postgresql: add replication + patroni Woodpecker secrets to provision-secrets step (ADR-0001 Phase 1)	2026-07-28 07:20:35 -07:00
AVB	efd90dac3e	Update ai/ai.yaml	2026-07-27 16:14:54 -07:00
admin	6789d2530a	vaultwarden: retry #8 -- trigger after fixing pipeline compile error (dollar-brace literal in comments)	2026-07-27 13:36:17 -07:00
admin	e0a6b7d233	deploy.yml: fix pipeline compile error 'missing closing brace' -- the documentation comment itself contained a literal dollar-brace sequence with invalid contents (dollar-brace CI_...), which Woodpecker's substitution engine parses even inside comments. Rewrote header to describe the rules in words without any literal dollar-brace sequences. Also removed backtick-quoted dollar-brace remnants from the vaultwarden case comment.	2026-07-27 13:35:13 -07:00
admin	9b3bba636c	vaultwarden: retry #7 -- trigger after fixing Woodpecker compile-time ${VAR} substitution root cause	2026-07-27 12:52:59 -07:00
admin	6757b0dfe1	deploy.yml: THE ACTUAL ROOT CAUSE FIX. Woodpecker substitutes ${VAR} at pipeline-compile time from its CI metadata map, where secrets do not exist -- so every braced reference to a secret-backed env var (SWARM_MANAGER_IP, all provisioning values) was silently blanked before the shell ever ran. Proven by debug output showing eval "val=\$$v" arriving as val=\$v ($$ unescaped to $) and lengths matching var NAMES. Fix: escape all runtime secret refs as $${VAR}; keep ${CI_PIPELINE_FILES} braced (compile-time is correct for CI metadata). Removes flawed debug step; adds header documenting the substitution rules.	2026-07-27 12:50:21 -07:00
admin	7a81bc73bd	deploy.yml: add temporary debug-secrets-all step declaring the identical 31-secret environment block as provision-secrets, printing byte length of every var (never values). Runs as its own step so it completes even when provision-secrets fails, telling us in one run whether all secrets are empty, only some, or only swarm_manager_ip.	2026-07-27 12:39:53 -07:00
admin	1a1068a154	vaultwarden: retry #6 -- re-trigger after fixing invalid 3DPRINT_DB_PASSWORD env var name	2026-07-27 12:13:05 -07:00
admin	332708ad41	deploy.yml: rename env var alias 3DPRINT_DB_PASSWORD -> PRINT3D_DB_PASSWORD in provision-secrets. Shell/env var identifiers cannot start with a digit (POSIX); this invalid key was likely poisoning the entire environment block for this step, explaining why SWARM_MANAGER_IP read empty specifically in provision-secrets/deploy/verify (which all declare 20-30 secrets) while working fine in the isolated 2-secret debug step. The underlying Woodpecker secret name (3dprint_db_password) is unchanged -- only the YAML alias and its $-reference were renamed.	2026-07-27 12:12:39 -07:00
admin	cfd735f14d	vaultwarden: retry #5 -- real end-to-end test now that repo is public and pipeline bugs are fixed	2026-07-27 11:57:30 -07:00
admin	01333893f1	deploy.yml: remove temporary debug-secrets step. Secret injection confirmed working (SSH_KEY=516 bytes, SWARM_MANAGER_IP=12 bytes) after repo was switched back to public. Root cause of the entire multi-day secret-provisioning saga: (1) CI_COMMIT_CHANGED_FILES renamed to CI_PIPELINE_FILES in Woodpecker 3.16, (2) --no-interpolate flag was misplaced, (3) dotfolders matched as bogus stack names, (4) private-repo flip broke Woodpecker's secret injection entirely. All four now fixed.	2026-07-27 11:57:02 -07:00
admin	d89cfbf055	deploy.yml: add temporary debug-secrets step (prints byte length of SSH_KEY and SWARM_MANAGER_IP only, never values) to determine whether secret injection is failing repo-wide or just for swarm_manager_ip. Repo has been switched back to public.	2026-07-27 11:54:54 -07:00
admin	6fd3018a8c	vaultwarden: retry #4 -- re-trigger after fixing --no-interpolate flag position and dotfolder false-match bugs	2026-07-27 11:03:58 -07:00
admin	c511b2b607	deploy.yml: fix two bugs surfaced now that CI_PIPELINE_FILES parsing actually works: (1) --no-interpolate was placed before the config subcommand instead of after, causing 'unknown flag' on any real folder-stack validate; (2) folder-detection regex matched dotfolders like .woodpecker as bogus stack names, causing SSH/SCP attempts with empty SWARM_MANAGER_IP. Also added explicit empty-var guards for SWARM_MANAGER_IP in provision-secrets/deploy/verify so future secret misconfig fails fast with a clear message instead of a cryptic ssh error.	2026-07-27 11:03:35 -07:00
admin	a9297397c7	vaultwarden: retry #3 -- re-trigger secret provisioning now that CI_PIPELINE_FILES parsing is fixed	2026-07-27 10:52:01 -07:00
admin	206f6ff103	deploy.yml: fix root cause - CI_COMMIT_CHANGED_FILES does not exist in Woodpecker 3.16; the correct variable is CI_PIPELINE_FILES (JSON array format, e.g. [\"a/b.yaml\",\"c.yaml\"]). Updated all 5 usages (validate, provision-secrets x2, deploy, verify, notify-success) to read CI_PIPELINE_FILES and strip [ ] \" before splitting on comma. Removes temporary debug-env step.	2026-07-27 10:51:37 -07:00
admin	2b6286a1c9	deploy.yml: add temporary debug-env step to diagnose CI_COMMIT_CHANGED_FILES not populating despite Woodpecker UI showing correct changed files. No logic changes to existing steps -- remove once root cause is found.	2026-07-27 10:31:10 -07:00
admin	9447b277a1	vaultwarden: remap DATABASE_URL secret to versioned name (vaultwarden_database_url_v2), same in-container target path. Fixes rotation being silently blocked by Swarm (cannot rm a secret referenced by a running service).	2026-07-27 08:50:43 -07:00
admin	6ff6260b2f	deploy.yml: fix vaultwarden secret provisioning (was bypassing shared helper, silently failing on rm-while-in-use). Provision under versioned name vaultwarden_database_url_v2; compose-side remap happens in a follow-up commit.	2026-07-27 08:50:16 -07:00
admin	71cb6a7e81	vaultwarden: retry secret re-provision trigger (previous run had empty changed-files)	2026-07-26 22:27:58 -07:00
admin	4a3eb44a62	vaultwarden: trigger secret re-provision (db -> postgresql alias)	2026-07-26 22:10:46 -07:00