Kubernetes-Native PostgreSQL HA
Why Patroni is the Right Choice for Cloud-Native PostgreSQL HA
Patroni is an open-source, production-grade template for PostgreSQL high availability orchestrated by distributed consensus (etcd, Consul, or Kubernetes API). It continuously manages primary leader election, automated health detection, and zero-data-loss replica promotion while exposing REST endpoints for proxy integration. Its integration with hardware watchdogs eliminates split-brain writes during network partitions. JusDB provides 24/7 Patroni DBRE, chaos-tested failover runbooks, and sub-15m P1 SLA.
Patroni uses a distributed configuration store (etcd, Consul, or ZooKeeper) for leader election and exposes a REST API for cluster management — making it the gold standard for automated PostgreSQL failover in Kubernetes and cloud environments.
What is Patroni?
Patroni is an open-source PostgreSQL high-availability template written in Python and maintained by Zalando. It wraps a PostgreSQL instance with a daemon that talks to a Distributed Configuration Store (etcd, Consul, ZooKeeper, or Kubernetes API) to elect a single primary, automatically promote a replica on primary failure, and prevent split-brain through DCS-backed locking.
First released in 2015, Patroni has become the de-facto standard for PostgreSQL HA in cloud and Kubernetes environments — it underpins the Zalando Postgres Operator and Crunchy Data PGO (CloudNativePG is a Patroni-free alternative that re-implements HA natively). Compared to repmgr (its main alternative), Patroni offers fully automatic failover; compared to Stolon, it works on bare-metal and VMs in addition to Kubernetes. Typical failover time is 30–45 seconds with default settings.
How Patroni Works
Patroni wraps PostgreSQL with a Python daemon that continuously communicates with a Distributed Configuration Store (DCS). The DCS holds the cluster state — which node is primary, the current timeline, and configuration. This eliminates split-brain by making leader election a distributed consensus problem.
DCS-backed Leader Election
Uses etcd, Consul, or ZooKeeper for distributed consensus. No more split-brain scenarios — only the node that holds the DCS lock is primary.
Automatic Failover in Seconds
When the primary fails, Patroni promotes the best replica within configurable TTL (default 30s). No manual intervention needed.
REST API & patronictl CLI
Manage switchover, promote, reinitialize, and pause cluster operations via REST API or the patronictl CLI — integrates with Kubernetes operators.
Fencing & Watchdog
Hardware or software watchdog support (e.g. /dev/watchdog) prevents the old primary from accepting writes after demotion — critical for data safety.
Configuration Management
Patroni stores PostgreSQL configuration in DCS so all cluster members use consistent settings. No config drift between nodes.
Kubernetes Operator Ready
Native integration with the Zalando PostgreSQL Operator and Crunchy Data PGO — Patroni is the standard HA backend for K8s-managed PostgreSQL (CloudNativePG re-implements this natively without Patroni).
Resilience Engineering
Patroni Production Failure Modes
Critical distributed consensus and failover failure modes mitigated by JusDB DBREs to prevent dual-primary writes, replication slot disk saturation, and spurious failover loops.
Watchdog Fencing Timeout & Demoted Leader Split-Brain
In the event of network partition or hypervisor stalls, an isolated primary node loses DCS lease heartbeat renewal. If the Linux hardware watchdog is misconfigured or disabled, the node continues accepting writes while the DCS cluster promotes a standby, creating divergent WAL timelines.
Enforce strict /dev/watchdog hardware or softdog device integration, configure watchdog_safety_margin = 5s, and validate node self-termination during simulated DCS isolation.
Replication Slot WAL Accumulation & Disk Exhaustion
When a standby fails or diverges, its persistent physical replication slot holds WAL segments on the primary. As writes continue, pg_wal expands unchecked until the host disk hits 100% capacity, freezing the primary database.
Configure max_slot_wal_keep_size limits in patroni dynamic configuration and instrument automated Prometheus alerts on replication slot lag bytes and disk saturation.
DCS Network Latency Jitter & Spurious Failover Loops
Transient network congestion or etcd disk fsync stalls exceed loop_wait (10s) and ttl (30s) lease timeouts, triggering unnecessary leader demotions, cascading failover elections, and client disconnection storms.
Size dedicated SSD/NVMe IOPS for etcd WAL disks, tune retry_timeout and loop_wait intervals for cross-AZ network latency, and pin etcd nodes to low-jitter network paths.
Cluster Telemetry
Production Patroni Diagnostic Runbooks
Non-blocking command-line and REST API introspection commands used by JusDB DBREs during cluster triage to verify quorum health, replication lag, and switchover readiness.
Audits cluster member roles, node host addresses, current timeline IDs, and standby replication byte lag directly from DCS state.
# Inspect live cluster membership, timeline IDs, and replica lag patronictl -c /etc/patroni/patroni.yml list # Inspect active dynamic configuration parameters stored in DCS patronictl -c /etc/patroni/patroni.yml show-config # Execute safe zero-data-loss manual switchover patronictl -c /etc/patroni/patroni.yml switchover --master pg-node-1 --candidate pg-node-2
Validates health status endpoints (/primary, /replica) utilized by proxies and load balancers to route live database traffic.
# Query REST API health on primary (returns HTTP 200 on leader)
curl -s -i http://localhost:8008/primary
# Inspect full JSON cluster state, WAL replay LSNs, and member lag
curl -s http://localhost:8008/cluster | jq '.members[] | {name: .name, role: .role, state: .state, lag: .lag, timeline: .timeline}'Patroni vs repmgr vs Cloud HA
All three tools solve PostgreSQL HA but with very different approaches. Patroni wins for cloud/Kubernetes environments. repmgr wins for simplicity. Stolon requires Kubernetes and has a smaller community.
| Evaluation Vector | JusDB Patroni DBRE | Repmgr / Scripted HA | Cloud Multi-AZ Default |
|---|---|---|---|
| Distributed Consensus & Leader Election | Production etcd/Consul DCS clustering with Raft leader leases, automated quorum healing, and zero risk of split-brain writes | Single-AZ or proprietary multi-AZ failovers with opaque orchestration logic and unconfigurable timeout policies | Custom keepalived / VIP scripts or repmgr witness nodes vulnerable to network partitions and dual-primary split-brain disasters |
| Hardware Watchdog & Node Fencing | Linux hardware/softdog integration with STONITH fencing; demoted primaries immediately kernel-reset if DCS heartbeat lease expires | Software-only fencing prone to thread freezes or delayed instance termination during hypervisor stalls | No hardware fencing; demoted former primaries continue accepting rogue writes from stale application connection pools |
| Synchronous Replication & Standby Sync | Automated synchronous_commit standby selection, dynamic quorum management, and replication lag threshold fencing | Asynchronous replication defaults where failover risks silent data loss (RPO > 0) during ungraceful primary crashes | Manually configuring synchronous_standby_names leading to total write stalls when any single replica experiences disk latency |
| Controlled Switchover & Maintenance | Safe patronictl switchover with automated LSN synchronization, zero data loss, sub-10 second failover window, and pgBackRest sync | Cloud provider maintenance windows requiring full database restarts and unpredictable switchover downtime | Manual promote commands with mismatched timeline IDs, broken standby replication streams, and prolonged downtime |
| REST API & Proxy Router Integration | Native Patroni REST API (/primary, /replica) integrated with HAProxy/PgBouncer for instantaneous application traffic redirection | DNS-based failover endpoints with TTL propagation delays causing 30-120 seconds of application connection errors | Homegrown health check scripts polling pg_isready that fail to differentiate between read-only standbys and primary writers |
| 24/7 Production DBRE & Sub-15m P1 SLA | Continuous DCS health inspection, replication slot leak alerting, senior DBRE on-call 24/7/365, and contractual <15m P1 response | General cloud support ticketing with multi-tier escalation delays during active database failover anomalies | Engineers attempting to debug etcd consensus raft logs and patronictl topology states at 2 AM during customer-facing outages |
When to Choose Patroni (vs Alternatives)
Choose Patroni when…
- • You run PostgreSQL on Kubernetes or any container platform
- • You need fully automatic failover with zero manual steps
- • You want a REST API to integrate cluster management into CI/CD
- • Your team already uses etcd or Consul for service discovery
- • You need multi-datacenter HA with leader election across DCs
- • You use Zalando Operator, Crunchy PGO, or CloudNativePG
Choose repmgr when…
- • You want minimal dependencies — no etcd/Consul required
- • You prefer manual switchover with a simple CLI
- • Your PostgreSQL runs on bare-metal or simple VMs
- • You need tight Barman backup integration out of the box
- • Team is not familiar with distributed systems or DCS tooling
- • Low-traffic PostgreSQL where manual intervention is acceptable
Patroni on Kubernetes
On Kubernetes, Patroni uses the Kubernetes API itself as the DCS — no separate etcd cluster is required. Cluster state lives in a ConfigMap or Endpoints object, and leader election uses the Kubernetes Lease primitive. Two established operators bundle Patroni as their HA backend; a third, CloudNativePG, re-implements the same behaviors natively without the Patroni daemon.
Zalando Postgres Operator
The original Patroni-based operator. Manages thousands of PostgreSQL clusters in production at Zalando. Best for teams that want minimal opinionation and direct YAML control of Patroni configuration.
Crunchy Data PGO
Patroni + pgBackRest + PgBouncer in one operator. Adds enterprise features: cross-cluster replication, multi-tenant deployments, and built-in monitoring.
CloudNativePG (EDB)
Newer operator that re-implements many Patroni-like behaviors natively. Best for teams already on EDB Postgres or who want CNCF-aligned tooling — Patroni-style HA without the Patroni daemon itself.
Production K8s pattern: 3-replica StatefulSet with PodDisruptionBudget = 1, pod anti-affinity on hostname, and HAProxy or PgBouncer in front of the Patroni REST API health endpoint (/master returns 200 only on the current primary).
Patroni REST API: Key Endpoints
Patroni exposes an HTTP REST API on port 8008 by default. These endpoints are the standard way to drive cluster operations from CI/CD, load balancers, and observability stacks.
| Endpoint | Purpose | Returns |
|---|---|---|
| GET /cluster | Full cluster topology + state | JSON of all members, lag, roles |
| GET /master | Health check for HAProxy / load balancers | 200 if primary, 503 otherwise |
| GET /replica | Health check for read-only routing | 200 if healthy streaming replica |
| GET /metrics | Prometheus-format metrics | timeline, lag, postmaster state |
| POST /switchover | Manual planned failover | Promotes specified replica |
| POST /restart | Rolling restart (zero-downtime) | Restarts node respecting role |
| POST /reinitialize | Reclone a divergent replica | Triggers pg_basebackup |
| PATCH /config | Update PostgreSQL/Patroni config in DCS | Propagates to all members |
All POST/PATCH endpoints require auth via restapi.authentication credentials defined in patroni.yml. For production, also enable HTTPS via restapi.certfile and restapi.keyfile.
Patroni Metrics & Observability
Patroni emits Prometheus-format metrics on /metrics. These six metrics catch the vast majority of PostgreSQL HA incidents before they cause an outage. Wire them into Grafana with alerts tuned to your TTL / loop_wait values.
patroni_master1 if the node is currently primary, 0 otherwise. Alert on unexpected leader changes.
patroni_replica1 if the node is a healthy streaming replica. Alert if a replica drops to 0 outside maintenance.
patroni_postgres_runningIs the PostgreSQL postmaster process up. Page on any 0 — Patroni itself running with PG down is the worst state.
patroni_xlog_replayed_locationWAL replay position. Compute lag against primary's xlog_location for replication-lag alerts.
patroni_pending_restart1 if a config change is awaiting restart. Alert if it sits >1 hour — usually a missed maintenance window.
patroni_dcs_last_seenUnix timestamp of last successful DCS interaction. Alert if it exceeds 2× ttl — DCS connectivity issue.
JusDB Patroni Implementation Service
We design and deploy production-grade Patroni clusters tuned to your infrastructure — bare-metal, VMs, EKS, GKE, or on-prem Kubernetes.
Architecture Design
3-node primary + replica topology, DCS sizing (etcd cluster vs single node), VIP or HAProxy/PgBouncer frontend layer design.
DCS Setup & Hardening
Deploy and secure etcd or Consul cluster, configure TTL and heartbeat intervals, test quorum loss scenarios.
Failover Testing & Runbooks
Simulate primary crash, network partition, and DCS unavailability. Document exact recovery steps and expected timelines.
Kubernetes Operator Integration
Configure Zalando PostgreSQL Operator or Crunchy PGO with Patroni as the HA backend, including PodDisruptionBudgets and pod anti-affinity rules.
Monitoring Integration
Patroni metrics exposed via /metrics endpoint, integrated into Prometheus + Grafana with alerting on leader changes, lag spikes, and DCS connectivity.
Ongoing Managed Operations
24/7 monitoring of cluster state, proactive replica lag alerts, coordinated version upgrades with zero downtime using Patroni's rolling restart.
Patroni PostgreSQL HA: Frequently Asked Questions
Ready to implement Patroni?
Our PostgreSQL HA specialists will design, deploy, and test your Patroni cluster — with runbooks, monitoring, and 24/7 support.