TiDB on Kubernetes
TiDB on Kubernetes
In short: Running TiDB on Kubernetes means using PingCAP's TiDB Operator and its TidbCluster CRD to manage four components: Raft-quorum PD, storage TiKV, and columnar TiFlash as StatefulSets with PersistentVolumeClaims, plus stateless MySQL-compatible TiDB Servers. The operator automates failover, horizontal scaling, rolling upgrades, and BR backups to object storage.
Deploy production-grade TiDB on Kubernetes with TiDB Operator, automated scaling, HTAP analytics, and MySQL-compatible distributed SQL for cloud-native workloads.
JusDB engineers production-grade TiDB on Kubernetes deployments utilizing the official TiDB Operator and custom CRD architecture. Certified DBREs configure dedicated local NVMe storage classes for TiKV and TiFlash, isolate Placement Driver quorum consensus, enforce PodDisruptionBudgets, and orchestrate zero-downtime rolling stateful upgrades backed by contractual 15-minute Sev-1 response SLAs and SOC 2 Type II compliance.
What we deliver
Comprehensive TiDB on Kubernetes Services
From TiDB Operator deployment to production HTAP workloads, we provide end-to-end TiDB on Kubernetes solutions for cloud-native distributed SQL environments.
TiDB Operator Deployment
TiDB Operator setup and configuration for automated TiDB cluster lifecycle management on Kubernetes
- TiDB Operator installation and CRD setup
- TidbCluster custom resource configuration
- RBAC and namespace isolation
- Multi-cluster operator management
Helm Chart Management
TiDB Helm chart customization, values tuning, and upgrade strategies for reproducible deployments
- TiDB Helm chart customization
- Values file tuning for production
- Chart version upgrade strategies
- GitOps integration with ArgoCD / Flux
TiDB Cluster on K8s
Full TiDB cluster deployment with PD, TiKV, TiDB Server, and TiFlash components on Kubernetes
- PD cluster setup and tuning
- TiKV storage node configuration
- TiDB Server SQL layer deployment
- TiFlash columnar engine setup
Storage & Persistence
PVCs, StorageClasses, and high-performance storage backends for TiKV and TiFlash on Kubernetes
- PVC and StorageClass configuration
- Local SSD and cloud block storage
- Separate storage tiers per component
- Volume expansion and snapshots
Monitoring & Observability
TiDB Dashboard, Prometheus, and Grafana on Kubernetes for full observability of TiDB clusters
- TiDB Dashboard deployment
- Prometheus and Grafana setup
- TidbMonitor CRD configuration
- Alertmanager rules and routing
Backup & Disaster Recovery
BR tool, Dumpling, and S3 backups for comprehensive TiDB data protection on Kubernetes
- BR (Backup & Restore) on K8s
- Dumpling logical export setup
- S3-compatible backup storage
- Point-in-time recovery with TiCDC
Component map
TiDB Architecture on Kubernetes
Understand how TiDB's distributed components map to Kubernetes primitives, managed by TiDB Operator for a fully automated, cloud-native deployment.
Metadata & Scheduling
PD (Placement Driver)
Cluster metadata and TiKV data scheduling
Distributed Storage
TiKV
Distributed transactional key-value store
SQL Layer
TiDB Server
MySQL-compatible stateless SQL engine
Columnar Analytics
TiFlash
Real-time columnar analytics engine
The case for K8s
Why Run TiDB on Kubernetes?
TiDB was built for cloud-native environments. Kubernetes provides the ideal platform for operating TiDB's distributed architecture with automated lifecycle management.
Cloud-Native Distributed SQL
TiDB was designed from the ground up as a cloud-native distributed SQL database. On Kubernetes, TiDB Operator manages all components declaratively, integrating seamlessly with your CI/CD pipelines and infrastructure-as-code workflows.
HTAP in a Single Platform
Combine OLTP and OLAP workloads in one database. TiKV handles transactions while TiFlash provides real-time columnar analytics, eliminating the need for separate ETL pipelines and analytics databases.
Elastic Horizontal Scaling
Scale TiDB components independently — add TiKV nodes for more storage, TiDB Servers for more SQL throughput, or TiFlash replicas for faster analytics. All without downtime or manual data rebalancing.
MySQL Compatibility
TiDB speaks the MySQL protocol, so existing MySQL applications, ORMs, and tools work with minimal changes. Migrate from MySQL to TiDB on Kubernetes and gain distributed SQL capabilities without rewriting your application.
TiDB on K8s Key Metrics
Methodology
Our TiDB on Kubernetes Implementation Process
A proven methodology for deploying production-ready TiDB on Kubernetes with comprehensive testing and validation.
Assessment & Planning
Evaluate your workload requirements, data volume, HTAP needs, and Kubernetes environment. Size PD, TiKV, TiDB Server, and TiFlash components based on throughput and storage targets.
Operator & Cluster Setup
Deploy TiDB Operator via Helm. Configure TidbCluster custom resources, StorageClasses for TiKV and TiFlash, networking, TLS, and resource requests for production readiness.
Migration & Validation
Migrate data from MySQL or other sources using TiDB Lightning or DM (Data Migration). Validate application compatibility, run failover tests, and benchmark OLTP and OLAP performance.
Production & Operations
Go live with TiDB Dashboard monitoring, Prometheus/Grafana alerting, automated BR backups, and runbooks. Provide team training on TiDB Operator management and 24/7 support.
How JusDB DBRE Kubernetes Engineering compares to alternative paths.
Operating stateful TiDB distributed SQL clusters on Kubernetes without dedicated DBRE operator expertise risks PD quorum loss, CSI persistent volume detachment deadlocks, and TiFlash OOM thrashing. Here is how our certified cloud-native engineering compares across core evaluation vectors:
| Evaluation Vector | JusDB DBRE | In-House DBA | Legacy Agency | Developer Generalist |
|---|---|---|---|---|
| TiDB Operator Architecture & TidbCluster CRD Tuning | Deploys official TiDB Operator with fine-tuned TidbCluster CRD manifests, granular component resource allocations, sysctl kernel parameters, and automated controller reconciliations across heterogeneous node groups. | Deploys generic community Helm charts without configuring component-specific CRD parameters, leading to operator reconciliation loops and misconfigured controller states. | Treats TiDB components as generic stateless workloads or standard StatefulSets, bypassing the TiDB Operator lifecycle automation and breaking automated self-healing. | Deploys all components in a single default namespace using basic manifests, lacking topology spread constraints and proper service account RBAC isolation. |
| Dedicated Local NVMe Storage Classes for TiKV/TiFlash | Provisions dedicated local NVMe storage classes via sig-storage-local-static-provisioner with tuned ext4/XFS mount options, delivering sub-millisecond Raft WAL sync latencies and high-IOPS RocksDB compaction. | Mounts standard cloud network block storage (EBS/gp3) with burstable IOPS, causing severe Raft commit latency spikes and cascading store disconnection events during write spikes. | Shares network-attached NFS or shared cluster storage across TiKV pods, violating multi-Raft consistency guarantees and triggering filesystem corruption warnings. | Uses ephemeral container root disks without persistent volume claims, risking catastrophic data loss upon pod restarts or worker node rescheduling. |
| Placement Driver (PD) Multi-AZ Quorum Isolation | Architects a dedicated 3-node or 5-node PD quorum spread across separate availability zones with pod anti-affinity, tuned Raft election ticks, and prioritized network routes to guarantee uninterrupted TSO timestamp generation. | Schedules all PD replicas on the same Kubernetes worker node or single failure domain, causing complete cluster-wide metadata paralysis when that single node reboots. | Runs an even number of PD replicas or mixes PD with heavy compute pods, leading to split-brain voting deadlocks during network partitions. | Exposes PD etcd ports directly across cluster networks without mutual TLS (mTLS) or network policies, compromising cluster consensus security. |
| PodDisruptionBudgets (PDB) & Anti-Affinity Hardening | Enforces strict PodDisruptionBudgets (maxUnavailable: 1), zone-level topologySpreadConstraints, and node anti-affinity for TiKV and PD pods, ensuring voluntary node drains and cluster auto-scaling never breach Raft quorum. | Omits PodDisruptionBudgets, allowing automated cluster upgrades or node autoscalers to terminate multiple TiKV stores simultaneously and causing region peer unavailabilities. | Configures soft anti-affinity that collapses during node shortages, placing multiple TiKV leader replicas onto a single overloaded Kubernetes hypervisor. | Ignores eviction tolerances and drain mechanics, resulting in sudden pod eviction cascades whenever worker nodes hit transient memory pressure. |
| Zero-Downtime Rolling Stateful Upgrades | Orchestrates phased rolling upgrades using TiDB Operator partition-based update strategies, automated TiKV region leader eviction prior to pod termination, and health check gates that guarantee zero transaction loss. | Performs uncoordinated kubectl rollout restarts, forcibly terminating TiKV stores with active leader leases and causing client connection aborts and multi-second query stalls. | Schedules full-cluster maintenance windows for minor patch updates, taking down business-critical HTAP applications for hours. | Updates container image tags simultaneously across all StatefulSets without verifying schema or storage engine backward compatibility, corrupting Raft state machines. |
| Cloud-Native Prometheus, Grafana & Chaos Mesh Testing | Integrates TidbMonitor with Prometheus operator, deploys curated Grafana dashboards with TiKV raft-engine and PD TSO alerts, and validates cluster resilience using automated Chaos Mesh fault-injection pipelines. | Relies on generic Kubernetes node-exporter metrics, lacking visibility into TiKV Coprocessor queue depths, RocksDB write stalls, or TiFlash delta-layer merges. | Installs ad-hoc logging agents without central telemetry aggregation, forcing reactive troubleshooting only after production outages have already impacted users. | Runs synthetic load tests in staging without chaos engineering or failure injection, discovering quorum split and CSI volume unmount deadlocks only in production. |
Kubernetes Failure Modes
Critical TiDB on Kubernetes Failure Modes We Eliminate
Running distributed stateful databases in containerized environments introduces CSI storage detachment deadlocks, multi-AZ quorum splits, and container cgroup memory limits. We engineer resilience into every layer to eliminate these production failure modes:
TiKV Persistent Volume CSI Detach Deadlock
During worker node drains or rescheduling events, cloud CSI storage drivers fail to release volume attachment locks for TiKV PersistentVolumes. TiKV pods remain in ContainerCreating indefinitely with VolumeAttachment deadlocks, causing Raft under-replication and stalling two-phase commit transactions.
JusDB provisions local NVMe storage classes via sig-storage-local-static-provisioner, deploys automated CSI attachment timeout reconcilers, and executes pre-drain volume unmount validation hooks before node evictions.
Placement Driver Pod Eviction Causing Cluster State Freeze
Kubernetes node autoscaler or cluster maintenance drains nodes hosting Placement Driver (PD) pods without respecting Raft consensus majorities. Dropping below PD quorum freezes TSO allocation, region metadata updates, and cluster-wide routing for all active client connections.
JusDB enforces strict PodDisruptionBudgets (maxUnavailable: 1) on PD StatefulSets, configures multi-AZ topologySpreadConstraints with hard podAntiAffinity, and scripts graceful leader transfer pre-stop lifecycle hooks.
TiFlash OOM Crash Due to Unconstrained Kubernetes Memory Limits
Massive vectorized analytical scans and complex cross-table joins consume unconstrained memory inside TiFlash containers. Exceeding cgroup memory limits triggers Linux kernel OOM killer termination, dropping columnar replicas and spilling expensive reporting queries back to TiKV.
JusDB defines explicit container resource requests and limits matched to node capacity, configures TiFlash memory tracker circuit breakers, enables spill-to-disk on high-speed NVMe scratch volumes, and tunes query_mem_limit to prevent runaway memory allocation.
Our DBREs execute non-blocking diagnostic commands across Kubernetes CRD controllers and internal TiDB cluster endpoints to audit topology, pod health, and storage capacity:
Audits custom resource controller status, PD/TiKV/TiDB/TiFlash pod distribution across Kubernetes worker nodes, node IP assignments, and pod phase conditions.
kubectl get tidbcluster,pods -n tidb-cluster -o wide
Inspects internal TiKV store capacity, available disk space percentage, leader region counts, and Up/Down operational status via PD controller.
kubectl exec -it -n tidb-cluster tidb-controller-manager -- pd-ctl -u http://pd:2379 store
FAQ
TiDB on Kubernetes — Frequently Asked Questions
Common questions about running TiDB on Kubernetes in production environments.
Ready to Run TiDB on Kubernetes?
Let our experts deploy and manage production TiDB on Kubernetes with TiDB Operator automation, HTAP workloads, and MySQL-compatible distributed SQL at scale.
Related TiDB Services
Explore more ways our TiDB experts can help with your database infrastructure.