Patroni architecture, YAML configuration & Kubernetes hub.
Patroni
Consensus & Fencing Stack (:8008)
etcd v3 Consensus
recommendedAtomic leader key lease management with strict ttl >= loop_wait + 2 * retry_timeout timing invariant.
REST Health API :8008
supportedBuilt-in HTTP health endpoints polled by HAProxy or Kubernetes Services to steer write and read connections.
Linux Watchdog Fencing
supportedKernel-level hardware or softdog timer that self-fences a stalled primary before a new leader is promoted.
pg_rewind Recovery
supportedAutomatic timeline divergence repair after an unplanned promotion so former leaders rejoin as replicas without full basebackup.
We engineer Patroni clusters on bare-metal Linux, AWS/GCP/Azure VMs, and Patroni Kubernetes (Spilo).
- P1: 15 min · P2: 30 min
- Initial-response targets under a 24x7 support agreement (P3: 60 min, P4: 90 min); 2 to 4 hours during covered business hours.
- Timer Invariant Formula
- ttl >= loop_wait + (2 * retry_timeout): prevents premature lease expiration during transient DCS or disk spikes.
- Consulting & Support
- Use this hub for patroni.yml and patronictl reference patterns, or visit our dedicated Patroni Consulting spoke for custom engagements.
Selected clients
Production Failure Modes
Six Patroni failure modes we resolve in production.
Patroni is only as reliable as its DCS quorum, lease timer math, and kernel watchdog fencing. Most production outages trace back to one of these six misconfigurations.
Split-brain dual-primary writes after a brief network partition
Requested timeline ... is not a child of this server's history
Linux `/dev/watchdog` fencing was left set to `mode: off`, or `pg_ctl stop -m fast` hung on D-state NFS/EBS I/O while etcd expired the leader key and promoted a replica.
Enable `/dev/watchdog` (`softdog` or hardware IPMI watchdog) with `mode: required`, verify `safety_margin`, and repair diverged timelines using `pg_rewind`.
Frequent false-positive failovers during nightly backups or compaction
demoting self because DCS is not accessible
etcd is co-located on the same NVMe/EBS disk as PostgreSQL WAL or backup streams, causing etcd fsync latency (`wal_fsync_duration_seconds`) to exceed `retry_timeout`.
Isolate etcd onto dedicated low-latency disks/nodes, verify `ttl: 30 / loop_wait: 10 / retry_timeout: 10`, and tune etcd `--heartbeat-interval` and `--election-timeout`.
Former primary refuses to rejoin cluster after failover
FATAL: could not receive data from WAL stream: timeline 4 diverges
`use_pg_rewind: true` is missing in `patroni.yml`, or PostgreSQL was initialized without `wal_log_hints = on` / `data_checksums` and lacks a superuser rewind credential.
Enable `use_pg_rewind: true`, `wal_log_hints = on`, and `remove_data_directory_on_rewind_failure: false`, then test `patronictl reinit` on staging.
All writes hang when a synchronous replica restarts
WARNING: canceling wait for synchronous replication
`synchronous_mode_strict: true` is enabled with only two nodes (1 leader + 1 replica), so losing the single replica blocks all commits indefinitely.
Deploy at least 3 PostgreSQL nodes when using `synchronous_mode: true` with `synchronous_node_count: 1`, or allow degraded async fallback where RPO permits.
Patroni Kubernetes pods flap during kube-apiserver rollouts
watched key expired / K8s Endpoints lock contention
Patroni Kubernetes deployments using `kubernetes.use_endpoints: false` (ConfigMaps) or overloaded API servers suffer watch drops when cluster-wide RBAC/webhook latency spikes.
Configure `use_endpoints: true` (or `Lease` objects in modern operators like Zalando/Spilo), tune `retry_timeout`, or back Patroni with a dedicated etcd v3 cluster.
Logical replication slots vanish after a Patroni switchover
ERROR: replication slot "cdc_slot" does not exist on new primary
Logical replication slots used by Debezium or DMS existed only on the old leader and were not synchronized to physical standbys before promotion.
Configure Patroni's `slots` permanent slot synchronization (`type: logical`) and PostgreSQL 17+ native failover slots (`sync_replication_slots = on`).
Response Targets & Incident Triage
Published P1 to P4 response targets and the first hour of a P1.
When etcd loses quorum or a diverged PostgreSQL timeline blocks replica recovery, you reach a named PostgreSQL/Patroni DBRE directly.
2 to 4 hours
Initial response target
Initial-response target during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30).
15 minutes
Initial response target
Initial-response target for P1 incidents where your service agreement includes 24x7 on-call.
Published Severity Matrix (under a 24x7 support agreement)
Production database down, primary unreachable, data corruption risk, or active failover event.
Severe latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.
Non-critical slow query regression, staging issue, backup warning, or replication check alert.
Configuration question, index advisory, planned schema review, or general operational request.
Our engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement. Every 24x7 response target above applies under a 24x7 support agreement and measures initial engineer response, not full incident resolution.
01 · Page
0 to 15 minutes (P1 target under a 24x7 support agreement)
Your alert reaches a senior PostgreSQL/Patroni DBRE directly, with no Tier-1 ticket queue.
02 · Triage
15 to 30 minutes
We run `patronictl -c /etc/patroni/patroni.yml list`, inspect etcd quorum health (`etcdctl endpoint status`), and verify PostgreSQL timeline history files.
03 · Mitigate
30 to 60 minutes
We stabilize leader lock ownership (pausing auto-failover via `patronictl pause` if DCS is flapping), restore HAProxy `:8008/primary` routing, and safely rewind diverged nodes.
04 · Review
Within 24 hours
We deliver a written incident report with corrected `patroni.yml` lease timers, `/dev/watchdog` verification, and Prometheus alert rules.
Architecture & Operational Scope
Six Patroni engineering pillars we design and manage.
From etcd v3 consensus rings on bare-metal Linux to Patroni Kubernetes StatefulSets with Spilo, we engineer every layer of the failover stack.
etcd v3 & Consul DCS Quorum Engineering
Design resilient 3-node and 5-node etcd v3 or Consul consensus clusters across availability zones so leader elections never suffer split-brain.
- Strict calibration of ttl, loop_wait, retry_timeout, and master_start_timeout.
- Dedicated etcd disk IOPS isolation and TLS mutual authentication.
Patroni Kubernetes & Spilo Orchestration
Deploy and harden Patroni inside Kubernetes StatefulSets using Zalando Postgres Operator (Spilo), Percona, or custom Helm charts.
- Endpoints vs Lease DCS lock tuning for Patroni Kubernetes clusters.
- PodDisruptionBudgets, anti-affinity rules, and preStop switchover hooks.
Linux Watchdog Fencing & pg_rewind
Eliminate dual-primary split-brain risks with kernel-level watchdog timers and automated timeline healing.
- Configuration of `/dev/watchdog` (softdog / IPMI) with `mode: required`.
- Automated `pg_rewind` validation with `wal_log_hints` and checksums enabled.
HAProxy + PgBouncer Routing Integration
Wire Patroni's `:8008` REST health API into HAProxy and PgBouncer so application pools follow promotions in under 2 seconds.
- `GET /primary` and `GET /replica?lag=16MB` HTTP health checks in HAProxy.
- Callback scripts (`on_role_change`) to reload or reconnect local PgBouncer pools.
Synchronous Replication & Failover Slots
Configure `synchronous_mode` for lossless RPO=0 commits and preserve logical CDC replication slots across primary switchovers.
- Quorum-based `synchronous_node_count` tuning across 3+ availability zones.
- Permanent physical and logical slot synchronization for Debezium and Kafka CDC.
Zero-Drama Switchovers & Major Upgrades
Execute planned `patronictl switchover` maintenance and low-downtime PostgreSQL major version upgrades.
- Pre-flight lag, checkpoint, and long-running transaction gates before switchover.
- Standby cluster cascading (`standby_cluster` stanza) for cross-region DR and pg_upgrade.
Interactive Quorum States & Reference YAML
How Patroni handles leader leases, watchdog fencing, and pg_rewind.
Compare a healthy 3-node Patroni + etcd quorum, an unfenced split-brain configuration, and a hardened watchdog-backed failover, followed by our reference patroni.yml and patronictl runbook.
Patroni + etcd v3 Consensus & Fencing State Simulator
Application and routing
pg-node-1 holds the `/service/pg-ha/leader` lease in etcd v3 (ttl=30s, loop_wait=10s); HAProxy routes :5000 writes to `/primary` and :5001 reads to `/replica?lag=16MB`.
pg-node-1 (Leader · TL 4)
HTTP 200 on :8008/primary
Holds etcd leader lock · watchdog active
pg-node-2 (Sync Standby · TL 4)
HTTP 200 on :8008/replica
sync_state = sync · 0 MB lag
pg-node-3 (Async Standby · TL 4)
HTTP 200 on :8008/replica
sync_state = potential · 1.2 MB lag
pg-node-1 holds the `/service/pg-ha/leader` lease in etcd v3 (ttl=30s, loop_wait=10s); HAProxy routes :5000 writes to `/primary` and :5001 reads to `/replica?lag=16MB`.
Application and routing
Without `/dev/watchdog` and with aggressive `ttl: 10s`, a storage stall freezes pg-node-1 while etcd promotes pg-node-2, risking dual-primary writes until HAProxy intervenes.
pg-node-1 (Stalled Old Leader)
D-state I/O freeze · no watchdog reset
Unfenced for 6s after lease expiry
pg-node-2 (Promoted to TL 5)
Acquires etcd leader key
Accepts new writes on Timeline 5
pg-node-1 Rejoin Attempt
WAL timeline divergence failure
use_pg_rewind was false -> manual rebuild
Without `/dev/watchdog` and with aggressive `ttl: 10s`, a storage stall freezes pg-node-1 while etcd promotes pg-node-2, risking dual-primary writes until HAProxy intervenes.
Application and routing
When pg-node-1 fails, `/dev/watchdog` self-fences the host 5 seconds before lease expiry; pg-node-2 promotes cleanly to Timeline 5 and pg-node-1 auto-rewinds on reboot.
pg-node-1 (Fenced & Rewound)
Watchdog triggered -> Rejoined on TL 5
pg_rewind repaired 12 MB WAL delta
pg-node-2 (New Leader · TL 5)
HTTP 200 on :8008/primary
Promoted with RPO=0 synchronous commit
pg-node-3 (Promoted to Sync Standby)
HTTP 200 on :8008/replica
Follows Timeline 5 automatically
When pg-node-1 fails, `/dev/watchdog` self-fences the host 5 seconds before lease expiry; pg-node-2 promotes cleanly to Timeline 5 and pg-node-1 auto-rewinds on reboot.
Production Configuration Specimen · Patroni 3.x / 4.x
Hardened patroni.yml with etcd v3 Quorum, Synchronous Commit & Watchdog
1. Reference /etc/patroni/patroni.yml (etcd3 + watchdog + pg_rewind)
/etc/patroni/patroni.yml
scope: pg-ha-prod
namespace: /service
name: pg-node-1
restapi:
listen: 0.0.0.0:8008
connect_address: 10.0.2.11:8008
etcd3:
hosts: ["10.0.2.21:2379", "10.0.2.22:2379", "10.0.2.23:2379"]
protocol: https
cacert: /etc/patroni/certs/etcd-ca.pem
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
synchronous_mode: true
synchronous_node_count: 1
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
wal_level: replica
wal_log_hints: "on"
max_wal_senders: 10
max_replication_slots: 10
hot_standby: "on"
watchdog:
mode: required
device: /dev/watchdog
safety_margin: 52. Operator switchover & maintenance runbook (patronictl)
patronictl CLI Operations
# 1. Inspect cluster topology, timeline numbers, and replication lag patronictl -c /etc/patroni/patroni.yml list # 2. Execute a planned zero-drama switchover to a healthy synchronous replica patronictl -c /etc/patroni/patroni.yml switchover pg-ha-prod \ --leader pg-node-1 \ --candidate pg-node-2 \ --scheduled now # 3. Pause auto-failover before OS kernel patching or etcd maintenance patronictl -c /etc/patroni/patroni.yml pause pg-ha-prod --wait # 4. Resume Patroni orchestration once all nodes report healthy patronictl -c /etc/patroni/patroni.yml resume pg-ha-prod
Shared Responsibility & Platforms
What your team owns and what JusDB owns.
Managed cloud databases automate much of the infrastructure layer, but your team still owns query performance, capacity, cost, and incident decisions. We operate Patroni on bare-metal Linux, AWS EC2/EKS, GCP Compute/GKE, Azure VMs/AKS, and hybrid multi-region topologies.
| Operational Layer | Customer Team | JusDB DBRE Team |
|---|---|---|
| Application connection strings & retry logic | Connect apps via HAProxy/PgBouncer VIP and handle brief ~5s reconnects | Tune TCP keepalives, PgBouncer pool timeouts, and HAProxy :8008 health probes |
| Patroni YAML & PostgreSQL parameters | Approve maintenance windows and GitOps infrastructure PRs | Author patroni.yml, tune DCS timers, configure pg_rewind, and manage dynamic DCS config |
| Consensus Store (etcd v3 / Consul / K8s DCS) | Provide 3 or 5 isolated VM/pod placement zones | Deploy, secure, monitor, and defragment etcd v3 or Patroni Kubernetes Endpoints/Leases |
| Kernel Watchdog & Split-Brain Fencing | Allow softdog kernel module or hardware IPMI watchdog access | Configure `/dev/watchdog` permissions, safety margins, and test fencing under load |
| Backup Integration (pgBackRest / Barman) | Provide S3/GCS/Azure Blob object storage buckets and IAM policies | Wire custom_bootstrap_script and replica creation callbacks to pgBackRest stanza restores |
Customer Boundary
What stays in your control
- Set business priorities, application requirements, and release plans
- Approve production changes, except routine actions you have pre-approved
- Own your cloud accounts, access policies, and vendor relationships
- Decide when maintenance can happen and how quickly service must recover
- Move the work in-house or to another provider after your agreement's notice period, with your data returned or deleted and the transition help your agreement sets out (Terms, section 8)
JusDB Execution Scope
What our DBREs execute for you
- Monitor database health, investigate alerts, and involve the right people
- Review query performance, replication, capacity, and backup health
- Carry out approved changes and update the operating instructions they affect
- Report completed work, open risks, and recommended next steps
HA Orchestrator Comparison & Engagement Paths
Patroni vs alternatives and how to engage JusDB.
Compare Patroni against repmgr, pg_auto_failover, and Stolon, then choose between our dedicated Patroni Consulting spoke, an ongoing PostgreSQL Support retainer, or a free HA architecture review.
| Evaluation Vector | JusDB Patroni DBRE | Repmgr / Scripted HA | Cloud Multi-AZ Default |
|---|---|---|---|
| Distributed Consensus & Leader Election | Production etcd/Consul DCS clustering with Raft leader leases, automated quorum healing, and leader-lease fencing designed to prevent split-brain writes | Single-AZ or proprietary multi-AZ failovers with opaque orchestration logic and unconfigurable timeout policies | Custom keepalived / VIP scripts or repmgr witness nodes vulnerable to network partitions and dual-primary split-brain disasters |
| Hardware Watchdog & Node Fencing | Linux hardware/softdog integration with STONITH fencing; demoted primaries immediately kernel-reset if DCS heartbeat lease expires | Software-only fencing prone to thread freezes or delayed instance termination during hypervisor stalls | No hardware fencing; demoted former primaries continue accepting rogue writes from stale application connection pools |
| Synchronous Replication & Standby Sync | Automated synchronous_commit standby selection, dynamic quorum management, and replication lag threshold fencing | Asynchronous replication defaults where failover risks silent data loss (RPO > 0) during ungraceful primary crashes | Manually configuring synchronous_standby_names leading to total write stalls when any single replica experiences disk latency |
| Controlled Switchover & Maintenance | Planned patronictl switchovers with replication-lag checks, synchronous mode where the recovery point requires it, rehearsed failovers, and pgBackRest backups | Cloud provider maintenance windows requiring full database restarts and unpredictable switchover downtime | Manual promote commands with mismatched timeline IDs, broken standby replication streams, and prolonged downtime |
| REST API & Proxy Router Integration | Patroni REST API health checks (/primary, /replica) drive HAProxy routing, with PgBouncer pooling connections behind it | DNS-based failover endpoints with TTL propagation delays causing 30-120 seconds of application connection errors | Homegrown health check scripts polling pg_isready that fail to differentiate between read-only standbys and primary writers |
| Monitoring & Incident Response | Continuous DCS health inspection, replication slot leak alerting, and senior DBREs on your incident bridge. Initial-response targets: 2 to 4 hours in covered business hours; 15 minutes for P1 incidents with 24x7 on-call. Targets, not resolution guarantees. | General cloud support ticketing with multi-tier escalation delays during active database failover anomalies | Engineers attempting to debug etcd consensus raft logs and patronictl topology states at 2 AM during customer-facing outages |
Patroni Architecture & Implementation Consulting
Fixed-scope engagement to design, deploy, or harden a Patroni + etcd v3 + HAProxy/PgBouncer cluster on Linux VMs or Kubernetes.
- Complete patroni.yml audit, DCS timer calibration, and softdog fencing setup.
- Integration with pgBackRest, HAProxy :8008 health checks, and PgBouncer.
- Game-day failover rehearsal with measured RTO/RPO and operator runbooks.
Ongoing PostgreSQL & Patroni Support
Named senior PostgreSQL DBREs operating your Patroni clusters and etcd consensus rings with published P1-P4 response targets.
- 15-minute P1, 30-minute P2, 60-minute P3, and 90-minute P4 response targets under a 24x7 support agreement.
- Planned `patronictl switchover` execution for OS and PostgreSQL minor patching.
- 24x7 monitoring of etcd fsync latency, replication slots, and timeline health where contracted.
Free Patroni & etcd Quorum Audit
Worried about false-positive failovers or split-brain edge cases? Have a senior PostgreSQL DBRE review your patroni.yml.
- Inspection of your bootstrap.dcs timers, watchdog config, and etcd topology.
- Verification of pg_rewind readiness and logical slot failover safety.
- 30-minute engineering readout with actionable YAML diffs.
Procurement and security
Ready for legal and security review.
- Mutual NDA before any access
- Named individual accounts only
- Read-only access by default
- ISO 27001 & SOC 2 Type II: aligned, certification in progress
Mutual NDA before any access
We sign a mutual NDA before viewing schemas, logs, slow-query captures, or cloud topology.
Least-privilege, named credentials
Engineers connect through your bastion, VPN, or SSM using named accounts with read-only permissions by default.
Change control & approval gates
Production DDL, parameter changes, and failovers follow your change-management workflow with rollback steps prepared first.
Named leadership & direct escalation
Engagements are led by named senior database reliability engineers in India with direct Slack, Teams, and PagerDuty channels.
Patroni Technical & Operational FAQ
Frequently asked Patroni questions.
Synced 1-to-1 with our structured FAQPage schema. Need help sizing an etcd v3 ring or configuring Patroni Kubernetes StatefulSets? Reach out to our PostgreSQL DBRE team.
Still have a question?
Ask it on a scoping call. No production access needed.
Book a 30-min scoping call (opens Google Calendar in a new tab)How is this Patroni Reference Hub different from JusDB's Patroni Consulting page?
How should Patroni's ttl, loop_wait, and retry_timeout timers be calculated?
Why does Patroni require `/dev/watchdog` (softdog) to prevent split-brain?
How does Patroni Kubernetes work compared to running Patroni with etcd v3 on VMs?
How does HAProxy know which Patroni node is the current primary?
How does Patroni compare to repmgr, Stolon, and CloudNativePG?
What are JusDB's SLA response targets for Patroni and PostgreSQL incidents?
Related Spokes, Tools & Reference Hubs
Continue exploring PostgreSQL HA & pooling.
Complete your PostgreSQL cluster architecture with HAProxy health routing, PgBouncer transaction pooling, and tuned postgresql.conf parameters.
Patroni Consulting Services
Fixed-scope architecture design, etcd quorum deployment, and failover game-day engagements.
PgBouncer Reference Hub
Pair Patroni failover routing with PgBouncer transaction pooling and prepared statement support.
HAProxy Database Routing Hub
Reference configs for polling Patroni's port 8008 /primary and /replica HTTP health endpoints.
repmgr Architecture Reference
Compare Patroni + etcd consensus against repmgrd witness-based standby cloning and promotion.
Stolon to Patroni Migration Guide
How to retire unmaintained Stolon keeper/sentinel clusters in favor of Patroni or CloudNativePG.
PostgreSQL Config Optimizer
Calculate shared_buffers, max_wal_size, and autovacuum parameters for your Patroni nodes.
Ready to Harden Your Patroni Cluster?
Get a senior PostgreSQL DBRE to audit your patroni.yml and etcd quorum.
Or request a free audit of one database instance, subject to availability: mutual NDA before read-only access, a prioritized report within one week after access is ready, and no obligation. Or jump straight to our commercial Patroni Consulting spoke at /tools/patroni/consulting.