Cassandra on Kubernetes fails in ways a stateless service never does. A worker node dies and the Cassandra pod sits in Pending with node(s) had volume node affinity conflict, because its data lives on a disk that no longer exists. A compaction spike pushes the container past its memory limit and Kubernetes reports OOMKilled. A cluster autoscaler drains two nodes in the same token range and LOCAL_QUORUM reads start failing. K8ssandra handles these cases: cass-operator manages each Cassandra datacenter as a set of StatefulSets, and k8ssandra-operator adds repairs, backups and multi-cluster support. This guide covers k8ssandra-operator v1.34.0 and cass-operator v1.33.1 with Apache Cassandra 5.0.9 or 4.1.12, the current releases as of November 2026.
- Use k8ssandra-operator v1.34.0, which installs cass-operator v1.33.1 and ships Reaper 5.1.0 and Medusa 0.31.1. Run Cassandra 5.0.9 or 4.1.12.
- cass-operator creates one StatefulSet per rack. Pin each rack to one zone with
nodeAffinityLabelsontopology.kubernetes.io/zone, and use a StorageClass withvolumeBindingMode: WaitForFirstConsumer. - Set
heap_initial_sizeandheap_max_sizeexplicitly and keep the container memory limit well above the heap. Off-heap memory and the management API share the limit. - Turn on Reaper with
autoSchedulingand schedule Medusa backups withMedusaBackupSchedule. A Medusa restore stops every pod in the datacenter. - Replace a dead node with a
CassandraTaskusing thereplacenodecommand, then repair if the node was down longer thanmax_hint_window(3 hours by default).
Scope and review date: November 2026. This post covers k8ssandra-operator v1.34.0 and cass-operator v1.33.1 running Apache Cassandra 4.1 and 5.0 on a managed Kubernetes service. Manifests, field names and defaults were checked against the K8ssandra documentation, the operators' source at those release tags, and the Apache Cassandra 5.0 documentation on 9 November 2026. They were not run in a lab, so apply them to a staging cluster first.
Which K8ssandra and cass-operator versions are current?
There are two operators. cass-operator owns the CassandraDatacenter resource (cassandra.datastax.com/v1beta1) and turns it into StatefulSets, Services and a PodDisruptionBudget. k8ssandra-operator adds the K8ssandraCluster resource (k8ssandra.io/v1alpha1), installs cass-operator through its Helm chart, deploys Reaper and Medusa, and can place datacenters in different Kubernetes clusters. Both are Apache 2.0 licensed and both had releases in the last two weeks.
| Component | Current release | What to know |
|---|---|---|
| k8ssandra-operator | v1.34.0 (changelog dated 30 October 2026) | Bundles cass-operator v1.33.1 (chart 0.68.1), Reaper 5.1.0 and Medusa 0.31.1. Removes Stargate support. |
| cass-operator | v1.33.1 (3 November 2026) | Fixes datacenter decommission when each DC runs in its own Kubernetes cluster |
| Apache Cassandra 5.0 | 5.0.9 (7 August 2026) | Latest GA line |
| Apache Cassandra 4.1 | 4.1.12 (7 August 2026) | Previous stable line |
The serverVersion field in the K8ssandraCluster API accepts Cassandra 3.11.x, 4.x.x and 5.x.x, and the cass-operator README recommends cass-operator 1.17.0 or newer for 4.1 and 5.0. Versions come from the v1.34.0 release notes, the cass-operator v1.33.1 release and the Apache Cassandra download page.
# cert-manager is a prerequisite in every Kubernetes cluster that runs the operator
helm repo add jetstack https://charts.jetstack.io
helm repo add k8ssandra https://helm.k8ssandra.io/stable
helm repo update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager --create-namespace --set crds.enabled=true
helm install k8ssandra-operator k8ssandra/k8ssandra-operator \
--namespace k8ssandra-operator --create-namespace --version 1.34.0Interpretation: the second install creates two Deployments, k8ssandra-operator and k8ssandra-operator-cass-operator. Pin --version so upgrades happen only when you choose. cert-manager's older installCRDs flag is deprecated in favor of crds.enabled.
How does the operator turn a K8ssandraCluster into pods?
You apply one K8ssandraCluster. The K8ssandraCluster controller, which runs only in the control-plane cluster, creates a CassandraDatacenter for each entry in spec.cassandra.datacenters. cass-operator then creates one StatefulSet per rack, named <cluster>-<dc>-<rack>-sts, and the StatefulSet controller creates the pods and their PersistentVolumeClaims (K8ssandra architecture).
From one K8ssandraCluster to pods and volumes
Each rack becomes its own StatefulSet, pinned to one zone
Lead: the PodDisruptionBudget cass-operator creates sets minAvailable to the datacenter size minus one, so a voluntary drain can take down only one Cassandra pod at a time.
Each pod runs a cassandra container, which also hosts the management API the operators call, and a server-system-logger container. Medusa adds a medusa sidecar and a medusa-restore init container. By default cass-operator allows one Cassandra pod per worker node, so six nodes need six schedulable workers.
kubectl get k8c,cassdc,sts,pdb -n k8ssandra-operator
kubectl get pods -n k8ssandra-operator -l cassandra.datastax.com/datacenter=dc1Interpretation: kubectl get k8c shows an ERROR column that reads None after a successful reconcile. Expect StatefulSets such as prod-dc1-r1-sts and pods at 3/3 ready with Medusa enabled.
How are seed nodes chosen?
You do not list seeds by hand. cass-operator picks one seed per rack or three per datacenter, whichever is more, labels those pods cassandra.datastax.com/seed-node=true, and publishes them through a headless Service named <cluster>-seed-service. Only ready pods keep the label, so if a seed goes down the operator labels another ready pod in the same rack. For nodes outside Kubernetes, such as a VM datacenter during a migration, list their addresses in spec.cassandra.additionalSeeds.
What does a production K8ssandraCluster manifest look like?
The manifest below runs Cassandra 5.0.9 in one datacenter of six nodes across three zones, with Reaper and Medusa. Field names match the v1.34.0 CRD schema. The deprecated heapSize field that older examples use is replaced by heap_initial_size and heap_max_size, and racks use nodeAffinityLabels because the rack zone field is deprecated.
apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
name: prod
namespace: k8ssandra-operator
spec:
cassandra:
serverVersion: "5.0.9"
datacenters:
- metadata:
name: dc1
size: 6
racks:
- name: r1
nodeAffinityLabels:
"topology.kubernetes.io/zone": us-east-1a
- name: r2
nodeAffinityLabels:
"topology.kubernetes.io/zone": us-east-1b
- name: r3
nodeAffinityLabels:
"topology.kubernetes.io/zone": us-east-1c
storageConfig:
cassandraDataVolumeClaimSpec:
storageClassName: cassandra-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Ti
resources:
requests:
cpu: "8"
memory: 32Gi
limits:
cpu: "8"
memory: 32Gi
config:
jvmOptions:
heap_initial_size: 8G
heap_max_size: 8G
cassandraYaml:
num_tokens: 16
reaper:
autoScheduling:
enabled: true
medusa:
storageProperties:
storageProvider: s3
region: us-east-1
bucketName: prod-cassandra-backups
prefix: prod
storageSecretRef:
name: medusa-bucket-keyInterpretation: the K8ssandraCluster name, prod, becomes the Cassandra cluster name. Each rack gets a required node affinity on its zone label. Cassandra places replicas on different racks, so with replication factor 3 a whole zone can fail and each token range keeps two replicas. num_tokens: 16 is already the default for Cassandra 4 and later; stating it makes it visible in review.
The admission webhooks reject later changes to num_tokens, the cluster name, datacenter names, rack names and rack order, and any part of the volume claim spec other than a size increase. Racks can only be added at the end of the list, and only with a big enough size increase to keep them balanced. Generated StatefulSet names must also stay within 60 characters.
Should Cassandra pods use local NVMe or network volumes?
Both work with cass-operator, but they fail differently. The cass-operator README says the StorageClass needs volumeBindingMode: WaitForFirstConsumer, so the volume is created in the zone where the scheduler places the pod and not in a random one. A network block volume such as Amazon EBS can only attach to a node in its own Availability Zone, which is another reason to map racks to zones (Amazon EBS volumes).
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cassandra-data
provisioner: ebs.csi.aws.com
parameters:
type: gp3
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: RetainInterpretation: Retain keeps the EBS volume when a PVC is deleted, for example by a node replacement, at the cost of orphaned volumes to clean up. Choose Delete if you rely on Medusa alone for recovery.
Local NVMe and network volumes for Cassandra pods
Switch storage types to see what changes when a worker node is lost
Local NVMe
| Question | Local NVMe (local PV) |
|---|---|
| Provisioning | Static PersistentVolumes only, created by hand or by a static provisioner |
| Where the data lives | On one worker node |
| Worker node lost | Pod cannot run; it stays Pending with a volume node affinity conflict |
| Recovery | replacenode task: new PVC on a spare worker, data streamed from replicas |
| Growing the volume | Fixed by the disk size; add nodes instead |
| I/O path | Device attached to the host, no network hop |
Network block volume
| Question | Network block volume (for example EBS gp3) |
|---|---|
| Provisioning | Dynamic, through the CSI driver and StorageClass |
| Where the data lives | On the storage service, in one zone |
| Worker node lost | Pod is rescheduled to another worker in the same zone and the PVC reattaches |
| Recovery | Usually none beyond hints or repair; the node rejoins with its data |
| Growing the volume | Allowed with allowVolumeExpansion and the cass-operator storage-change annotation |
| I/O path | Over the network; IOPS and throughput set by the volume type |
Trade-off: local NVMe turns every worker failure into a full data stream from other replicas. Network volumes turn it into a reschedule, at the cost of a network hop on every read and write.
The Kubernetes documentation is direct about local volumes: if the node becomes unhealthy, the local volume becomes inaccessible and the pod using it cannot run (Kubernetes volumes: local). That is acceptable for Cassandra, which keeps replicas on other racks, as long as you keep a spare worker with a free disk in each zone. To grow a network volume, add cassandra.datastax.com/allow-storage-changes: "true" to the datacenter's metadata.annotations and raise the storage request. cass-operator refuses any shrink.
How should you size memory, CPU and JVM heap for Cassandra pods?
Cassandra uses a lot of memory outside the Java heap: off-heap memtables, bloom filters, compression metadata, network buffers, and the page cache that serves SSTable reads. The management API also runs in the cassandra container, with its own heap set by mgmtAPIHeap. The container's memory limit has to cover the heap and all off-heap use, and whatever is left over is all the page cache gets. Kubernetes enforces that limit with an OOM kill, and the pod then reports OOMKilled (Kubernetes resource management).
Set the heap explicitly rather than relying on a calculated default. In 5.0, cassandra-env.sh sets the heap to half the memory reported by free -m, with an upper cap. Inside a container, free usually reports the worker node's memory, not the pod limit, so the calculated heap can be larger than the pod. Set heap_initial_size and heap_max_size to the same value.
The 8G heap inside a 32Gi limit in the manifest above is an example, not a rule. The right ratio depends on your workload and compaction strategy, so measure heap after GC and container working set under production-like load.
# Which pods were OOM-killed, and how close is each pod to its limit?
kubectl get pods -n k8ssandra-operator \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[?(@.name=="cassandra")].lastState.terminated.reason}{"\n"}{end}'
kubectl top pod -n k8ssandra-operator --containers | grep cassandraInterpretation: any OOMKilled in the first output means the limit is too small for heap plus off-heap use, or the heap is too large. kubectl top needs metrics-server; a working set that sits close to the limit is a warning even before the first kill.
Kubernetes gives its strictest QoS class only when every container in the pod, including the logger and the Medusa sidecar, has CPU and memory requests equal to their limits (Pod QoS classes). Pods in that class are the last to be evicted under node memory pressure.
How do repairs with Reaper and backups with Medusa work?
cass-operator does not schedule repairs or backups; K8ssandra adds both. With a reaper block, the operator deploys Reaper for each datacenter (default deploymentMode: PER_DC) and keeps its state in a reaper_db keyspace. With autoscheduling on, Reaper creates a schedule for every non-system keyspace and follows keyspace changes. The default repairType is AUTO, which means incremental repair on Cassandra 4 and later. Every table must finish a repair within gc_grace_seconds; the K8ssandra repair guide suggests one repair per week as a starting point. The Reaper UI password is in the prod-reaper-ui Secret. Anyone with UI access can pause or delete schedules, so keep the Service internal.
Medusa runs as a sidecar and uploads SSTables to object storage (s3, s3_compatible, s3_rgw, azure_blobs or google_storage). Backups are differential by default, so unchanged SSTables are not uploaded again. Schedule them and the purge separately:
apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaBackupSchedule
metadata:
name: dc1-nightly
namespace: k8ssandra-operator
spec:
backupSpec:
backupType: differential
cassandraDatacenter: dc1
cronSchedule: "30 1 * * *"
---
apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaBackupSchedule
metadata:
name: dc1-purge
namespace: k8ssandra-operator
spec:
backupSpec:
cassandraDatacenter: dc1
cronSchedule: "0 0 * * *"
operationType: purgeInterpretation: each run creates a MedusaBackupJob and a MedusaBackup named after the schedule plus a timestamp. Retention comes from maxBackupAge and maxBackupCount in storageProperties (0 means unlimited). kubectl get medusabackup should show SUCCESS with every node finished.
A MedusaRestoreJob stops every Cassandra pod in the datacenter, and the medusa-restore init container restores data when the pods start again (K8ssandra backup and restore). Rehearse it on a copy of the cluster and time it before you rely on it.
How do you scale and replace a failed Cassandra node?
To scale, change size. cass-operator adds or removes nodes evenly across racks, bootstraps one node at a time, and runs nodetool cleanup on the existing nodes after a scale-up (K8ssandra scaling). Keep size a multiple of the rack count. On scale-down, the operator checks that the remaining nodes have room for the data, sets the Valid condition to false with NotEnoughSpaceToScaleDown if they do not, and otherwise runs nodetool decommission before removing the pod.
A dead node is replaced with a CassandraTask using the replacenode command. The task drains the pod if Cassandra is still up, deletes its PVCs and the pod, and cass-operator starts the new pod as a replacement for the old host. Use it after losing a worker with local NVMe, or for a node with corrupt data.
Replacing a Cassandra node lost with its worker
Six steps; step 4 deletes the old node's volumes, so confirm the node is really gone first
-
Confirm the node is down in the ring
Run nodetool from a healthy pod. The lost node shows as DN.
kubectl exec -n k8ssandra-operator \ prod-dc1-r2-sts-0 -c cassandra -- \ nodetool -u "$CASS_USER" -pw "$CASS_PASS" status -
Read why the pod is not running
A local PV bound to a deleted worker gives a volume node affinity conflict in the events.
kubectl describe pod prod-dc1-r1-sts-1 \ -n k8ssandra-operator | tail -n 20 -
Check capacity in the same zone
The rack is pinned to one zone, so a worker with a free disk must exist there.
kubectl get nodes \ -l topology.kubernetes.io/zone=us-east-1a -
Create the replacenode task
The task deletes the pod's PVCs and the pod. This is the point of no return for the old volume.
kubectl apply -n k8ssandra-operator \ -f replace-prod-dc1-r1-sts-1.yaml -
Watch the replacement stream
The task completes when the new pod has joined. Large nodes take hours to stream.
kubectl get cassandratask -n k8ssandra-operator -w -
Repair if the outage was long
If the node was down longer than max_hint_window before replacement, run a Reaper repair of its keyspaces.
kubectl port-forward -n k8ssandra-operator \ svc/prod-dc1-reaper-service 8080
Main risk: replacing the wrong pod deletes a healthy node's data. Match the pod name to the DN host ID before step 4.
apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
name: replace-prod-dc1-r1-sts-1
namespace: k8ssandra-operator
spec:
datacenter:
name: dc1
namespace: k8ssandra-operator
jobs:
- name: replace-r1-1
command: replacenode
args:
pod_name: prod-dc1-r1-sts-1Interpretation: the task needs pod_name or rack, and the validator rejects a replace with neither. The Apache Cassandra documentation says you must repair a replaced node if it was down longer than max_hint_window before replacement, because it missed writes (Replacing a dead node). The 5.0 default is max_hint_window: 3h.
How do you run Cassandra across multiple datacenters or regions?
In K8ssandra, a datacenter can live in another Kubernetes cluster. Install the operator in every cluster, register each remote cluster with a ClientConfig in the control-plane cluster, and set k8sContext on the datacenter. The control-plane cluster runs the K8ssandraCluster controller; the K8ssandra documentation recommends not running a Cassandra datacenter in it. Pod IPs must be routable between all clusters, which on a managed service usually means VPC-native pod networking with peered networks.
apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
name: prod
namespace: k8ssandra-operator
annotations:
k8ssandra.io/dc-replication: '{"dc2": {"orders": 3, "events": 3}}'
spec:
cassandra:
serverVersion: "5.0.9"
datacenters:
- metadata:
name: dc1
k8sContext: us-east
size: 6
- metadata:
name: dc2
k8sContext: eu-west
size: 6Interpretation: when you add dc2 to a running cluster, the annotation sets each user keyspace's replica count in the new DC, and it must list every user keyspace. The operator then rebuilds the new DC from the existing one, one node per rack at a time unless you raise rebuild.maxConcurrentRebuilds (Add a datacenter). The webhook rejects renaming a DC, or adding and removing DCs in one change.
How do you upgrade the operators and move from Cassandra 4.1 to 5.0?
Upgrade the operators and Cassandra in separate changes. Since k8ssandra-operator 1.18, an operator upgrade no longer restarts running clusters by itself. Clusters that need new images or settings get a RequiresUpdate condition, and you apply the change by editing the spec or by setting the k8ssandra.io/autoupdate-spec annotation to once (K8ssandra upgrade notes). The Helm chart runs a CRD upgrader job on upgrade.
helm repo update
helm upgrade k8ssandra-operator k8ssandra/k8ssandra-operator \
--namespace k8ssandra-operator --version 1.34.0
kubectl annotate k8c prod -n k8ssandra-operator k8ssandra.io/autoupdate-spec=onceInterpretation: annotate in a maintenance window. It triggers a rolling restart if the pod spec changes, and the operator then removes the annotation.
A Cassandra upgrade is a change to serverVersion. cass-operator restarts pods one at a time on the new image. For 4.1 to 5.0, read the Cassandra 5.0 NEWS.txt first. It removes DateTieredCompactionStrategy and the dateOf and unixTimestampOf functions, drops Java 8, and changes the default disk_access_mode to mmap_index_only. It also adds storage_compatibility_mode, which defaults to CASSANDRA_4, so an upgraded node writes nothing that 4.x cannot read until you move it on:
- Take a full Medusa backup, let a repair cycle finish, and change any DTCS tables to TWCS.
- Set
serverVersion: "5.0.9"and let the rolling restart finish. You can still roll back to 4.1 binaries at this point. - After a burn-in period, set
storage_compatibility_mode: UPGRADINGunderconfig.cassandraYamland roll again. Nodes now move to the 5.0 storage format, so from here going back to 4.1 means restoring a backup. - Set it to
NONEand roll a third time. - Rewrite older SSTables with an
upgradesstablestask.
apiVersion: control.k8ssandra.io/v1alpha1
kind: K8ssandraTask
metadata:
name: prod-upgradesstables
namespace: k8ssandra-operator
spec:
cluster:
name: prod
template:
jobs:
- name: upgradesstables
command: upgradesstablesInterpretation: a K8ssandraTask creates one CassandraTask per datacenter and runs them one DC at a time (K8ssandra cluster tasks). The webhooks do not block a serverVersion downgrade, so review that field in every pull request.
What should you watch on Cassandra on Kubernetes in production?
Watch the operator's view and Cassandra's view together. Each can look healthy while the other shows a problem.
Operator state and ring state for one datacenter
Illustrative output shaped like kubectl and nodetool 5.0; host IDs shortened
Read both: None and Ready mean the operators have nothing left to do; six UN nodes with even load across three racks mean the ring agrees.
The superuser credentials are in the prod-superuser Secret, under the username and password keys. Beyond the ring state, track these signals:
- CassandraDatacenter conditions such as
ScalingUp,ReplacingNodesandRequiresUpdatethat stay true longer than the operation should take. - Restarts and OOM kills on the
cassandracontainer, and Pending pods, which usually mean no free worker or local volume in that zone. - Reaper and Medusa: repairs finishing within
gc_grace_seconds, and the age of the newest successfulMedusaBackup. - Free disk per PVC, because compaction needs headroom.
Setting telemetry.prometheus.enabled: true makes the operator create a ServiceMonitor per datacenter, if the Prometheus Operator CRDs are installed.
Frequently asked questions
Should I use K8ssandra or cass-operator on its own?
Use k8ssandra-operator for most production clusters. cass-operator alone manages the CassandraDatacenter, StatefulSets, seeds, scaling and node replacement, but it does not schedule repairs or backups. k8ssandra-operator v1.34.0 installs cass-operator v1.33.1 and adds Reaper for repairs, Medusa for backups, multi-cluster datacenters and cluster-wide tasks. Choose cass-operator alone only if you already run repairs and backups another way.
Which Cassandra versions does K8ssandra support?
The K8ssandraCluster API accepts Cassandra 3.11.x, 4.x.x and 5.x.x through the serverVersion field. As of November 2026 the current Apache releases are 5.0.9 and 4.1.12, both from 7 August 2026, and management API images exist for both. The cass-operator README recommends cass-operator 1.17.0 or newer for Cassandra 4.1 and 5.0.
How do I replace a dead Cassandra node in K8ssandra?
Create a CassandraTask with the replacenode command and the pod_name of the dead node. The task drains the pod if it is still running, deletes its PVCs and the pod, and cass-operator starts a new pod that replaces the old host and streams its data from replicas. If the node was down longer than max_hint_window, 3 hours by default, run a repair afterwards.
How do I upgrade Cassandra 4.1 to 5.0 with K8ssandra?
Take a Medusa backup, replace any DateTieredCompactionStrategy tables, then change serverVersion to 5.0.9 and let cass-operator restart pods one at a time. Cassandra 5.0 starts in storage_compatibility_mode CASSANDRA_4, so rollback is still possible. After a burn-in period, roll through UPGRADING and then NONE, and finish with an upgradesstables task.
Key takeaways
- Run k8ssandra-operator v1.34.0 with cass-operator v1.33.1 and Cassandra 5.0.9 or 4.1.12, and pin the Helm chart version.
- One StatefulSet per rack, one rack per zone, one Cassandra pod per worker, and a StorageClass with
WaitForFirstConsumer. - Set the JVM heap explicitly with room under the container limit for off-heap memory, page cache and the management API.
- Enable Reaper autoscheduling and Medusa backup and purge schedules, and rehearse a restore, which stops the whole datacenter.
- Plan node replacement before you need it, especially on local NVMe, and move from 4.1 to 5.0 through
storage_compatibility_modeone step at a time.
Run Cassandra on Kubernetes with JusDB
JusDB's Cassandra on Kubernetes team can review your K8ssandraCluster and CassandraDatacenter manifests, rack-to-zone mapping, storage classes and heap sizing, and set up Reaper repair schedules and Medusa backup and restore checks. We can also plan a node replacement runbook, a 4.1 to 5.0 upgrade, or a move from VMs into Kubernetes as a new datacenter.
Bring your manifests, Kubernetes and Cassandra versions, node sizes and current repair and backup setup. Your team approves every production change.