Finding
Moving from unpinned EBS gp3 pods to NodeConfig NVMe RAID-0 + static cpuset cut p99 by 94%
The cluster originally ran on m6i.4xlarge nodes with gp3 PVCs and cpuManagerPolicy: none, suffering 65ms Seastar reactor stalls. Migrating to i4i.4xlarge local NVMe formatted via NodeConfig with static CPU pinning stabilized p99 at 1.2ms.
1. NodeConfig CRD: Assemble local NVMe SSDs into XFS RAID-0 & run iotune
scylla-nodeconfig.yaml · scylla.scylladb.com/v1alpha1
apiVersion: scylla.scylladb.com/v1alpha1
kind: NodeConfig
metadata:
name: scylla-nvme-pool
spec:
placement:
nodeSelector:
scylla.scylladb.com/node-type: scylla-nvme
tolerations:
- key: scylla-clusters
operator: Equal
value: "dedicated"
effect: NoSchedule
disableOptimizations: false
localDiskSetup:
raids:
- name: nvme-raid0
type: RAID0
raid0:
devices:
nameRegex: ^/dev/nvme[1-9]n1$
mounts:
- device: nvme-raid0
mountPoint: /mnt/raid-disks/nvme-raid0
fsType: xfs
unsupportedOptions: ["prjquota"]
filesystems:
- device: nvme-raid0
fsType: xfs2. Production ScyllaCluster CRD: Pinned QoS (cpuset), 3-AZ racks & S3 backup
scylla-cluster.yaml · scylla.scylladb.com/v1
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
name: prod-scylla
namespace: scylla
spec:
version: 6.1.1
agentVersion: 3.3.2
cpuset: true
automaticOrphanedNodeCleanup: true
datacenter:
name: us-east-1
racks:
- name: us-east-1a
members: 2
storage:
capacity: 3200Gi
storageClassName: scylladb-local-xfs
resources:
requests:
cpu: "14"
memory: "110Gi"
limits:
cpu: "14"
memory: "110Gi"
repairs:
- name: weekly-repair
cron: "0 2 * * 6"
parallel: 2
backups:
- name: daily-s3-backup
cron: "0 1 * * *"
location: ["s3:prod-scylla-backups-us-east-1"]
retention: 7






