Free audit

View Audit Scope
ScyllaDB on K8sscylla-operator · NodeConfig XFS NVMe RAID-0 · Static cpuset

ScyllaDB on Kubernetes Consulting & DBRE Services

JusDB architects, deploys, and operates production ScyllaDB clusters on Kubernetes across Amazon EKS, Google GKE, Azure AKS, and bare-metal OpenShift. Our database reliability engineers configure the official Scylla Operator (scylla.scylladb.com/v1), automate local NVMe RAID-0 XFS provisioning via NodeConfig and scylla-csi-driver, enforce kubelet static CPU Manager core pinning (--cpuset), and manage Scylla Manager repair and S3 backup schedules.

The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Or request a free audit of one database instance.

  • NDA before access
  • Read-only by default
  • Written scope before work starts

ScyllaDB on Kubernetes DBRE controls

Select a K8s subsystem to inspect

Kubelet Static CPU Manager & --cpuset

Pinned QoS required

ScyllaDB's Seastar reactor threads poll CPU cores continuously. Running ScyllaDB in a Burstable pod or without kubelet cpuManagerPolicy: static causes Linux CFS context switching and severe tail-latency stalls.

We configure EKS/GKE/AKS worker kubelets with static CPU Manager policy and Pinned QoS integer requests == limits so each Seastar shard owns an exclusive physical vCPU.

Inspect ScyllaCluster & NodeConfig YAML

Verified against scylla.scylladb.com/v1 Operator, NodeConfig XFS RAID-0, and Seastar cpuset specs.

Kubernetes stacks covered
Scylla Operator (ScyllaCluster & NodeConfig), Scylla Manager, EKS (i4i/i3en), GKE (Z3/N2D), AKS (Lsv3), and OpenShift
Initial-response targets
15 minutes for a critical (P1) incident under a 24x7 support agreement; 2 to 4 hours in business hours. See details
Coverage hours
Business hours (03:30–12:30 UTC) by default; 24x7 on-call available under a 24x7 support agreement

Selected clients

  • Innoviti
  • Blitz
  • Freecharge
  • Nango
  • Recochain
  • Screen Facts
  • Nexora
  • WeCredit

Production failure modes

Common ScyllaDB on Kubernetes failures we resolve

ScyllaDB's C++ Seastar engine is built to own raw NVMe queues and physical CPU cores; scheduling it onto generic Kubernetes nodes with network EBS volumes and unpinned CPUs destroys its latency guarantees.

ScyllaDB on Kubernetes production symptoms, root causes, and first diagnostic actions
Kubernetes / ScyllaDB signalWhat is usually happeningWhere we start
Reactor Stalls (CFS Throttled)Seastar logs reactor stalled for 45 ms despite low cluster QPSThe ScyllaCluster pod runs with fractional CPU requests/limits or on a worker node where kubelet cpuManagerPolicy is set to 'none', allowing the OS scheduler to preempt Seastar polling threads.Enable kubelet cpuManagerPolicy: static on a dedicated ScyllaDB NodePool, set integer CPU requests == limits (Pinned QoS), and verify --cpuset pinning in pod logs.
iotune / EBS I/O BottleneckCompaction backlogs grow and p99 read latency exceeds 25ms on K8sScyllaCluster PVCs were provisioned on standard network-attached EBS gp2/gp3 or ext4 storage instead of local NVMe SSDs formatted with XFS Direct I/O.Deploy Scylla Operator's NodeConfig CRD + local-csi-driver on NVMe instance types (i4i/i3en/Z3/Lsv3) to assemble XFS RAID-0 and generate calibrated io_properties.yaml.
Dead Ephemeral NVMe NodeEC2 / GCE worker node terminates and takes local NVMe data with itBecause local NVMe SSDs are ephemeral to the physical hypervisor, an instance termination permanently destroys that node's local SSTables and stalls the StatefulSet pod.Configure Scylla Operator automatic dead-node replacement (or run nodetool removenode / replace-address-first-boot) so a fresh pod streams replicas from the other 2 AZ racks.
Port 19042 SNAT MismatchShard-aware drivers fall back to multi-hop coordinator routingKubernetes ClusterIP Services or CNI source-NAT (SNAT) rewrite the client's ephemeral TCP source port, breaking ScyllaDB's source-port-to-shard modulo calculation.Connect clients via headless member Services (direct Pod IP routing without SNAT) or use ScyllaDB's modern port 19042 shard-id connection negotiation.
OOMKilled on Pod Memorykubelet kills scylla container with Exit Code 137 during repair or compactionScyllaDB's --memory flag was not lowered below resources.limits.memory, leaving zero cgroup headroom for the scylla-manager-agent sidecar, page tables, and kernel socket buffers.Let scylla-operator calculate --memory and --reserve-memory automatically from Pinned QoS limits, reserving 1.5-2 GiB for sidecars and OS overhead.
Raft Quorum Lost on DrainClusterAutoscaler or Karpenter drains two racks simultaneouslyUnconstrained node consolidation evicts two ScyllaDB pods holding replicas for the same token range or Tablet before streaming repair completes.Apply karpenter.sh/do-not-disrupt: 'true' on ScyllaDB pods, isolate racks onto dedicated per-AZ node groups, and enforce Scylla Operator's PodDisruptionBudget.

Technical scope

What does our ScyllaDB on Kubernetes service include?

We engineer the full stack from cloud NVMe NodePools and NodeConfig XFS RAID-0 arrays up to ScyllaCluster CRDs, Scylla Manager S3 backups, and shard-aware client routing.

  • Scylla Operator (scylla.scylladb.com/v1) architecture

    We deploy, audit, and upgrade the official Scylla Operator, ScyllaCluster CRDs, and Scylla Manager controllers across EKS, GKE, AKS, and OpenShift.

    • Architect multi-AZ rack topologies (RF=3 across 3 racks) with GitOps-ready manifests
    • Configure ScyllaDB 6.x Raft-managed Tablets and rack-by-rack rolling upgrades

    scylla.scylladb.com/v1 · ScyllaCluster · ScyllaOperatorConfig · ArgoCD

    ScyllaDB consulting & schema review
  • NodeConfig local NVMe RAID-0 & XFS CSI storage

    ScyllaDB is engineered for local NVMe speed. We automate ephemeral NVMe disk discovery, RAID-0 striping, XFS formatting, and Seastar iotune on Kubernetes.

    • Deploy NodeConfig CRDs and scylla-local-csi-driver across i4i, i3en, Z3, and Lsv3 node pools
    • Automate scylla_io_setup (iotune) calibration and IRQ interrupt tuning on worker nodes

    NodeConfig CRD · local-csi-driver · XFS RAID-0 · scylla_io_setup

  • Kubelet static CPU Manager & Seastar cpuset pinning

    We eliminate Seastar reactor stalls on Kubernetes by aligning kubelet CPU Manager policies, NUMA topology, and Pinned QoS pod resources.

    • Configure kubelet cpuManagerPolicy: static and topologyManagerPolicy: single-numa-node
    • Calibrate ScyllaDB --smp, --cpuset, and --memory reservation alongside sidecar containers

    cpuManagerPolicy: static · Pinned QoS · --cpuset · --reserve-memory

    ScyllaDB performance tuning
  • Shard-aware networking (Port 19042) & CNI tuning

    We ensure client drivers reach the exact Seastar vCPU shard owning each partition without kube-proxy iptables bottlenecks or SNAT port mangling.

    • Configure headless member Services and exposeOptions (BroadcastOptions) for direct pod routing
    • Validate Cilium eBPF, AWS VPC CNI, and GKE Dataplane V2 with shard-aware port 19042

    Port 19042 · Headless Service · Cilium eBPF · TokenAware + ShardAware

  • Automated node recovery, Scylla Manager & S3 backups

    When an ephemeral NVMe cloud node dies, Kubernetes and Scylla Operator must replace the member cleanly. We automate repair, backup, and node replacement.

    • Configure Scylla Manager for automated weekly anti-entropy repair and S3/GCS backups
    • Test dead-node replace-address-first-boot and Tablet/vnode rebuild workflows

    Scylla Manager · sctool repair · sctool backup (S3/GCS) · replaceNode

  • Scylla Monitoring Stack & 24x7 K8s DBRE support (where contracted)

    We integrate the official Scylla Monitoring Stack into kube-prometheus-stack for per-shard visibility and back it with 24x7 on-call coverage where contracted.

    • Deploy ServiceMonitors for Seastar reactor stalls, LSA cache misses, and compaction queues
    • Provide P1-P4 incident response and day-2 Operator upgrades under a 24x7 support agreement

    Scylla Monitoring Stack · Prometheus · Grafana · PagerDuty alerts

    ScyllaDB 24x7 support (where contracted)

Shared responsibility & response targets

Where basic Operator manifests stop and what JusDB engineers run

Installing scylla-operator via Helm does not configure kubelet static CPU Manager, assemble NVMe RAID-0 arrays, or wire S3 backup recovery on your cloud node groups.

Comparison between basic Scylla Operator setups and JusDB Kubernetes DBRE engineering
Kubernetes subsystemWhere default setups stopWhat JusDB engineers run
Seastar CPU scheduling in KubernetesSchedules ScyllaDB pods onto default worker nodes (cpuManagerPolicy: none), causing constant CFS preemption and reactor stalls.Configures dedicated NodePools with kubelet static CPU Manager, single-NUMA alignment, and Pinned QoS integer CPUs.
Storage architecture (NVMe vs EBS/PD)Mounts generic network-attached EBS gp3 or standard CSI volumes that throttle Seastar Direct I/O and compaction.Provisions local NVMe RAID-0 formatted with XFS via Scylla Operator NodeConfig and local-csi-driver, calibrated with iotune.
Ephemeral NVMe node failure recoveryTreats local NVMe loss as an unrecoverable PVC failure or leaves dead nodes stranded in the ring.Architects 3-AZ rack replication (RF=3) so Scylla Operator automatically replaces a failed NVMe node and streams data from surviving racks.
Shard-aware driver routing (Port 19042)Exposes ScyllaDB behind a standard ClusterIP Service that SNATs source ports and breaks single-hop shard routing.Configures headless member Services and VPC-native CNI routing so drivers maintain direct per-core connections on port 19042.
Anti-entropy repair & S3 backupsDeploys the ScyllaCluster CRD alone without Scylla Manager, leaving tombstones unrepaired and backups unscheduled.Deploys and tunes Scylla Manager with IRSA / Workload Identity for automated S3/GCS SSTable backups and throttled repairs.
Karpenter & Cluster Autoscaler protectionAllows node autoscalers to drain ScyllaDB NVMe nodes at random, triggering expensive cluster-wide data streaming.Locks ScyllaDB nodes with karpenter.sh/do-not-disrupt, dedicated taints, and rack-aware PodDisruptionBudgets.

Published initial-response targets by severity

Standard business hours: 03:30–12:30 UTC

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees.

SeverityInitial responseDefinitionChannel & coverage
P1 · Critical15 minutesProduction database down, primary unreachable, data corruption risk, or active failover event.24x7 on-call (under your service agreement) · Direct PagerDuty, Opsgenie, or phone page plus incident bridge
P2 · High30 minutesSevere latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.24x7 on-call (under your service agreement) · On-call alert plus shared Slack or Teams channel
P3 · Medium60 minutesNon-critical slow query regression, staging issue, backup warning, or replication check alert.24x7 on-call (under your service agreement) · Shared Slack or Teams channel or support ticket
P4 · Low90 minutesConfiguration question, index advisory, planned schema review, or general operational request.24x7 on-call (under your service agreement) · Shared Slack or Teams channel or support ticket

Our engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement.

Initial-response targets, not resolution guarantees. The target for your engagement is set in your service agreement.

Interactive ScyllaCluster rack simulator

How Scylla Operator handles cpuset pinning and NVMe node loss

Compare a 3-AZ rack-pinned ScyllaCluster on local NVMe RAID-0 against an unpinned EBS deployment and an automated ephemeral NVMe node replacement.

ScyllaDB on Kubernetes Operator & NVMe rack simulator

Select a state to inspect kubelet cpuset pinning, XFS local-csi storage, and LOCAL_QUORUM continuity.

Headless Member Services (Port 19042), Scylla Operator & 3-AZ Racks

Shard-aware clients connect via headless Services on port 19042 directly to the owning Seastar vCPU shard; LOCAL_QUORUM reads/writes complete in < 1.5ms.

RF=3 across 3 AZs · Static cpuset · XFS Direct I/O
  • scylla-us-east-1a-0 (Rack A · i4i.4xlarge)

    Pinned 14 CPU (static cpuset) · 3.75TB NVMe RAID-0

    UN (Up/Normal) · 0ms reactor stalls

  • scylla-us-east-1b-0 (Rack B · i4i.4xlarge)

    Pinned 14 CPU (static cpuset) · XFS local-csi

    UN (Up/Normal) ·LOCAL_QUORUM replica

  • scylla-us-east-1c-0 (Rack C + Scylla Manager)

    Pinned 14 CPU · Scheduled S3 backup + repair

    UN (Up/Normal) · Raft quorum 3/3 healthy

Each ScyllaCluster rack maps to a dedicated AZ NodePool with kubelet static CPU Manager, NodeConfig XFS RAID-0 local NVMe, and RF=3 replication.

Representative production deliverable

How a ScyllaOperator NodeConfig and ScyllaCluster audit is delivered

Every engagement delivers the NodeConfig NVMe RAID-0 manifest, the Pinned QoS ScyllaCluster CRD, and the Scylla Manager S3 backup configuration.

Sample deliverable · NodeConfig XFS RAID-0 NVMe & 3-AZ ScyllaCluster CRDEKS 1.31 · i4i.4xlarge · ScyllaDB 6.1

Finding

Moving from unpinned EBS gp3 pods to NodeConfig NVMe RAID-0 + static cpuset cut p99 by 94%

The cluster originally ran on m6i.4xlarge nodes with gp3 PVCs and cpuManagerPolicy: none, suffering 65ms Seastar reactor stalls. Migrating to i4i.4xlarge local NVMe formatted via NodeConfig with static CPU pinning stabilized p99 at 1.2ms.

LOCAL_QUORUM write p9922.4ms to 1.2ms
Reactor stalls > 25ms410 / hr to 0

1. NodeConfig CRD: Assemble local NVMe SSDs into XFS RAID-0 & run iotune

scylla-nodeconfig.yaml · scylla.scylladb.com/v1alpha1

apiVersion: scylla.scylladb.com/v1alpha1
kind: NodeConfig
metadata:
  name: scylla-nvme-pool
spec:
  placement:
    nodeSelector:
      scylla.scylladb.com/node-type: scylla-nvme
    tolerations:
      - key: scylla-clusters
        operator: Equal
        value: "dedicated"
        effect: NoSchedule
  disableOptimizations: false
  localDiskSetup:
    raids:
      - name: nvme-raid0
        type: RAID0
        raid0:
          devices:
            nameRegex: ^/dev/nvme[1-9]n1$
    mounts:
      - device: nvme-raid0
        mountPoint: /mnt/raid-disks/nvme-raid0
        fsType: xfs
        unsupportedOptions: ["prjquota"]
    filesystems:
      - device: nvme-raid0
        fsType: xfs

2. Production ScyllaCluster CRD: Pinned QoS (cpuset), 3-AZ racks & S3 backup

scylla-cluster.yaml · scylla.scylladb.com/v1

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: prod-scylla
  namespace: scylla
spec:
  version: 6.1.1
  agentVersion: 3.3.2
  cpuset: true
  automaticOrphanedNodeCleanup: true
  datacenter:
    name: us-east-1
    racks:
      - name: us-east-1a
        members: 2
        storage:
          capacity: 3200Gi
          storageClassName: scylladb-local-xfs
        resources:
          requests:
            cpu: "14"
            memory: "110Gi"
          limits:
            cpu: "14"
            memory: "110Gi"
  repairs:
    - name: weekly-repair
      cron: "0 2 * * 6"
      parallel: 2
  backups:
    - name: daily-s3-backup
      cron: "0 1 * * *"
      location: ["s3:prod-scylla-backups-us-east-1"]
      retention: 7

Ways to work together

How to engage JusDB for ScyllaDB on Kubernetes

Start with a free read-only audit of your ScyllaCluster and NodeConfig manifests, scope a turnkey NVMe Operator deployment, or put your K8s ring under ongoing DBRE coverage.

Scoped K8s DBRE projectThis page

Deploy, harden, or upgrade Scylla Operator on NVMe.

A fixed-scope engagement delivering Terraform/GitOps NVMe NodePools, NodeConfig + ScyllaCluster CRDs, Scylla Manager S3 backups, and chaos node-loss testing.

Common K8s scopes agreed in writing first

  • ScyllaCluster & NodeConfig CRDs
  • Local NVMe RAID-0 XFS CSI
  • Static CPU Manager & cpuset
  • Port 19042 shard-aware routing
  • Dead-node replace & S3 backups
Free K8s ScyllaDB audit

One ScyllaCluster CRD and node pool, audited read-only.

We inspect your kubelet CPU Manager config, NodeConfig NVMe setup, ScyllaCluster resource limits, and repair/backup schedules, then deliver a written report within one week.

What we check on your Kubernetes cluster

  • cpuManagerPolicy: static & Seastar --cpuset pinning
  • Local NVMe RAID-0 XFS vs network EBS/PD bottlenecks
  • 3-AZ rack quorum, port 19042 SNAT & S3 backups
Ongoing K8s DBRE & support

Named engineers covering your Kubernetes ScyllaDB ring.

Proactive Scylla Monitoring Stack alerting, Operator version upgrades, S3 backup restore drills, and incident response under a 24x7 support agreement.

  1. Page on-callavailable under a 24x7 support agreement
  2. 15 minP1 initial-response target
  3. Written RCAafter every P1

Kubernetes platforms and NVMe cloud node families we support

Evaluation Vector
JusDB DBRE
In-House DBALegacy AgencyDeveloper Generalist
ScyllaDB Operator Architecture & ScyllaCluster CRD TuningArchitects the official Scylla Operator deploying ScyllaCluster and NodeConfig CRDs; tunes host sysctl parameters (fs.aio-max-nr, fs.file-max, vm.max_map_count), hugepages, and lifecycle hooks for rock-solid Kubernetes lifecycle management.Deploys manual StatefulSets without operator reconciliation, lacking automated NodeConfig OS tuning and experiencing kernel lockups under heavy IOPS.Treats ScyllaDB as a generic Cassandra pod deployment using basic bash wrappers that bypass Kubernetes admission webhooks and validation controllers.Applies third-party community Helm charts without configuring operator CRDs, leading to failed pod rollouts and unmanaged schema mutations.
Seastar C++ Shard-per-Core CPU Pinning & Core IsolationEnforces Guaranteed Quality of Service (QoS) with Kubernetes static CPU manager policy, pinning Seastar reactor threads 1:1 to dedicated physical cores and isolating IRQ balancing to eliminate context switching and P99 latency spikes.Allocates Burstable QoS with fractional CPU shares, triggering aggressive CFS quota throttling and severe Seastar reactor execution stalls.Relies on standard container multi-threading defaults, allowing adjacent noisy-neighbor pods to steal CPU cycles and destroy NoSQL tail latency.Assigns fractional CPU requests (e.g., 3.5 cores), causing Seastar core calculation crashes and unhandled thread initialization panics.
Local NVMe RAID-0 StorageClasses & Direct I/O SchedulingProvisions dedicated local NVMe drives formatted with XFS in RAID-0 via DirectPV or local-static-provisioner, tuning Seastar Direct I/O (O_DIRECT) and automated iotune benchmarks, targeting sub-millisecond p99 I/O latency.Mounts remote network-attached block storage (e.g., EBS gp3) without iotune calibration, causing write compaction backpressure and disk stalls.Uses default ext4 storage classes over standard cloud disks, causing buffer cache thrashing and breaking Seastar async userspace I/O scheduling.Mounts unbacked ephemeral container storage or dynamic storage without volume retention policies, risking total data loss upon pod restart.
Multi-AZ Rack-Aware Topology & Replication Factor MappingMaps ScyllaDB logical racks 1:1 to Kubernetes availability zones using topologySpreadConstraints and ScyllaOperator rack definitions, ensuring NetworkTopologyStrategy replicas survive whole-zone datacenter outages.Deploys all Scylla pods across random worker nodes in a single AZ, creating single points of failure that cause total cluster downtime during zone incidents.Configures Cassandra SimpleStrategy replication with 1 rack spanning multiple zones, causing cross-AZ egress cost explosions and uncoordinated quorum writes.Ignores zone affinity rules, allowing the Kubernetes scheduler to co-locate multiple shard replicas on the same underlying physical host node.
Scylla Manager Automated Repair & Rolling UpgradesIntegrates Scylla Manager for scheduled non-blocking row-level repairs, automated backup snapshots to S3/GCS, and coordinates rolling upgrades with automated gossip convergence verification.Schedules manual nodetool repair cronjobs that oversaturate disk I/O, causing latency spikes and client timeout cascades during production hours.Performs manual in-place container restarts without draining client connections, triggering read/write consistency failures and token range gaps.Triggers simultaneous pod updates using default RollingUpdate strategies, causing immediate quorum loss across Cassandra CQL client drivers.
Cloud-Native Prometheus, Syscall Profiling & Node Exporter TelemetryDeploys the Scylla Monitoring Stack with Prometheus ServiceMonitors, Grafana per-shard dashboards, eBPF/bcc syscall tracing, and custom alerting for reactor stalls and compaction backlog.Relies on generic Kubernetes container CPU/memory metrics, lacking visibility into Seastar shard reactor stalls, cache hit ratios, or SSTable compaction queues.Installs third-party APM agents inside the database container, introducing JVM/Python background threads that disrupt Seastar CPU core isolation.Operates blind with basic container liveness probes, failing to detect silent node gossip partition splits until client requests time out.

For procurement and platform security teams

Mutual NDA first, namespace-scoped RBAC, and GitOps delivery

All NodeConfig, ScyllaCluster, and Terraform NodePool changes are delivered via pull requests into your repositories under mutual NDA and least-privilege RBAC.Raguraman, Engineering Manager · Allen D, Lead Database SRE
  • Mutual NDA
  • Written statement of work
  • Scoped K8s RBAC
  • GitOps PR workflow
  • NDA before access

    Signed before inspecting any ScyllaCluster manifests, NodeConfigs, or Prometheus metrics.

  • Written statement of work

    Target Kubernetes clusters, deliverables, chaos drill criteria, and fees agreed in writing.

  • Read-only RBAC by default

    Scoped read-only Kubernetes Role bindings and CQL system inspection grants.

  • Manifests & runbooks you keep

    Every ScyllaCluster CRD, NodeConfig spec, and NVMe recovery runbook stays in your Git repository. Meet the JusDB team.

JusDB's security practices are aligned with ISO 27001 and SOC 2 Type II, and JusDB is pursuing both certifications (see Terms, section 12). Read the stated status.

Technical FAQ

ScyllaDB on Kubernetes common questions

Answers on ephemeral local NVMe safety, NodeConfig XFS RAID-0, kubelet static CPU Manager, Exit 137 memory reservation, and port 19042 routing.

Still have a question?

Ask it on a scoping call. No production access needed.

Book a 30-min scoping call (opens Google Calendar in a new tab)

Can ScyllaDB run reliably on Kubernetes using ephemeral local NVMe SSDs?

Yes - in fact, local NVMe SSDs (such as AWS i4i/i3en, GCP Z3/N2D Local SSD, and Azure Lsv3) are the recommended production architecture for ScyllaDB on Kubernetes. Because ScyllaCluster distributes 3 replicas of every partition across 3 Availability Zone racks (RF=3), durability is provided by cross-AZ Raft/quorum replication plus Scylla Manager S3/GCS backups rather than slow network-attached EBS volumes. If a worker node with ephemeral NVMe fails, the remaining 2 racks continue serving LOCAL_QUORUM traffic while Scylla Operator automatically replaces and rebuilds the lost node.

How does Scylla Operator's NodeConfig CRD prepare worker nodes and NVMe drives?

The Scylla Operator NodeConfig CRD runs a privileged DaemonSet on your dedicated ScyllaDB worker node pool. It automatically discovers attached local NVMe devices, assembles them into a Linux mdadm RAID-0 array, formats the array with XFS, runs scylla_io_setup (iotune) to benchmark hardware IOPS/bandwidth into io_properties.yaml, isolates NIC/disk hardware interrupts from Seastar CPU cores, and exposes the storage to pods via scylla-local-csi-driver.

Why is kubelet static CPU Manager policy mandatory for ScyllaDB on Kubernetes?

ScyllaDB's C++ Seastar engine uses a thread-per-core reactor model where each thread spins and polls its assigned vCPU core, memory pool, and I/O queues without kernel locks. If Kubernetes schedules ScyllaDB under the default CFS quota scheduler (cpuManagerPolicy: none), threads migrate across cores and suffer context-switch pauses ('Reactor stalled for X ms'). Enabling cpuManagerPolicy: static on the worker kubelet and setting integer CPU requests == limits (Pinned QoS) pins each Seastar shard exclusively to a physical CPU core.

How do you prevent Kubernetes OOMKilled (Exit Code 137) crashes on ScyllaDB pods?

Each ScyllaCluster pod runs both the main scylla container and sidecar containers (such as scylla-manager-agent), plus kernel page tables and socket buffers inside the pod's cgroup memory boundary. We configure Pinned QoS memory limits and ensure Scylla Operator reserves 1.5 to 2.5 GiB via --reserve-memory (or explicit container limits) so ScyllaDB's LSA allocator never pushes total pod RSS over the kubelet OOM threshold.

How do shard-aware client drivers connect to ScyllaDB pods inside and outside Kubernetes?

Inside the same Kubernetes cluster, applications connect via the headless member Services created by Scylla Operator, which return individual Pod IPs so shard-aware drivers can establish direct per-core connections on port 9042 and shard-aware port 19042 without kube-proxy SNAT rewriting source ports. For clients outside the Kubernetes cluster, we configure ScyllaCluster exposeOptions (BroadcastOptions with VPC-routable Pod IPs, NLB per member, or hostNetworking).

How do you handle ScyllaDB version upgrades and rack scaling on Kubernetes?

Scylla Operator performs rolling upgrades one pod (and one rack) at a time. Before terminating a pod, it drains client connections and flushes memtables via nodetool drain, waits for the upgraded pod to rejoin the ring as Up/Normal (UN), and verifies Raft/quorum health before moving to the next rack. With ScyllaDB 6.x Tablets enabled, scaling racks up or down streams Tablets in parallel across shards in minutes.

How are repairs and S3/GCS backups automated for ScyllaDB on Kubernetes?

We deploy Scylla Manager alongside Scylla Operator and bind the pod ServiceAccount to AWS IAM Roles for Service Accounts (IRSA / Pod Identity) or GCP Workload Identity. Scylla Manager coordinates token-range anti-entropy repairs across all pods automatically and uploads deduplicated SSTable snapshots directly from each pod's NVMe drive to S3 or GCS.

What are your response targets for ScyllaDB on Kubernetes incidents?

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees.

Next step

Talk to an engineer about your ScyllaDB on Kubernetes cluster

Tell us which Kubernetes platform and NVMe instance family you run with scylla-operator. The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Or request a free audit of one database instance, subject to availability: mutual NDA before read-only access, a prioritized report within one week after access is ready, and no obligation.

NDA before access · read-only by default · written scope first

Related ScyllaDB Services

Explore more ways our ScyllaDB experts can help with your database infrastructure.