A row you deleted last month is back. Or a repair that used to take an hour now runs for two days and ends with Repair command #12 finished with error. On Apache Cassandra 4.1 and 5.0, both problems come from the same place: anti-entropy repair is not running often enough, or not finishing. Cassandra does not repair data on a schedule unless you configure it. Hints cover short outages only. Any replica that misses a delete and is not repaired within gc_grace_seconds can bring the deleted data back. This guide covers full and incremental repair, -pr, subranges and parallelism, scheduling with Cassandra Reaper or the auto repair scheduler in Cassandra 5.0.8, and failed repairs.
- Every node must finish a repair within
gc_grace_seconds(default 864000, which is 10 days). Otherwise deleted data can come back. The Apache docs suggest repairing every node at least every 7 days. - Incremental repair is the default on 4.1 and 5.0 and only compares unrepaired SSTables. Since 4.0 it isolates its SSTables before it builds Merkle trees, which fixed the over-streaming problems of 3.x. Still run a full repair (
--full) every 1 to 3 weeks. - Run
nodetool repair -pron every node in every datacenter, or split the ring into subranges with-stand-et. - Reaper 5.1.0 (26 October 2026) schedules segmented repairs with back-pressure. Cassandra 5.0.8 and later also include the CEP-37 auto repair scheduler. It is off by default, and the JVM flag that enables it cannot be turned off again.
Scope and review date: November 2026. This post covers Apache Cassandra 4.1 (latest 4.1.12) and 5.0 (latest 5.0.9), and Reaper 5.1.0. Commands, defaults and messages were checked on 9 November 2026 against the Apache Cassandra documentation, the 4.1 and 5.0 source branches and the Reaper documentation. They were not run in a lab, so rehearse them on a staging cluster first.
Why does Cassandra need repair at all?
Cassandra keeps writing when a replica is down, so replicas drift apart. Hints replay missed writes only for outages shorter than max_hint_window (3 hours by default), and the repair documentation calls them best effort. Anti-entropy repair compares replicas and copies the differences.
Deletes make this urgent. A delete writes a tombstone, which compaction may drop once it is older than gc_grace_seconds. A replica that missed the tombstone still holds the old value, and once the tombstone is gone elsewhere, repair copies that value back. The tombstones page calls this deleted data a zombie.
-- cqlsh: the grace period for every table in one keyspace
SELECT table_name, gc_grace_seconds
FROM system_schema.tables
WHERE keyspace_name = 'shop';Interpretation: 864000 seconds is the 10-day default. The shortest value in the keyspace sets your repair deadline. Every node must complete a repair of that table in less time than that, with slack for failed runs.
Do not just start it and run repair. The tombstones it missed may already be purged elsewhere, so its old data would be repaired back into the cluster. The usual fix is to wipe the node and bring it back as a replacement, so it streams current data from the other replicas.
What does nodetool repair actually do?
nodetool repair repairs only the token ranges of the node it runs on. For each range and table, that node (the coordinator) asks every replica for a Merkle tree, a tree of hashes built by a validation compaction that reads every SSTable in the range. It compares the trees and streams only the ranges whose hashes differ.
One repair session, from Merkle trees to streaming
The validation read in step 2 happens on every replica, even when nothing is out of sync
Where the cost is: validation is a full read of the range on each replica, and streaming adds writes and compaction. Incremental repair cuts the validation input to unrepaired SSTables only.
Four cassandra.yaml settings, the same in 4.1 and 5.0, bound that cost:
# cassandra.yaml (5.0 defaults shown as comments in the shipped file)
# repair_session_space: # default 1/16 of heap, minimum 1 MiB
# concurrent_merkle_tree_requests: 0 # 0 = no limit
# concurrent_validations: 0 # below 1 = same as concurrent_compactors
compaction_throughput: 64MiB/s # also throttles validation compactionsInterpretation: with a smaller repair_session_space, each Merkle tree leaf covers more data, so one changed row streams a bigger range. The shipped file says to raise it, or use subrange repair, against over-streaming, and to lower it under heap pressure.
Full or incremental repair: which should you run?
Both 4.1 and 5.0 run incremental repair when you type nodetool repair, and full repair with --full. Incremental repair has a bad history: before 4.0, SSTables could be compacted between Merkle tree building and anticompaction. They were then never marked repaired, and the next repair over-streamed them (CASSANDRA-9143). The Last Pickle's write-up of the 4.0 change describes the fix. Cassandra now anticompacts first and moves the candidate SSTables into a pending repair pool that only that session uses. It sets their repairedAt timestamp only when the whole session succeeds. Full repair no longer anticompacts and no longer marks anything repaired.
Full and incremental repair on Cassandra 4.1 and 5.0
Switch types to see what each one reads, rewrites and marks
Full repair
| Aspect | Full |
|---|---|
| Command | nodetool repair --full |
| Data compared | All SSTables in the range |
| Anticompaction | None since 4.0 |
| Marks data as repaired | No |
Snapshots with -seq or -dcpar | Yes, on every replica |
| Catches corruption in already repaired data | Yes |
| Reaper parallelism | Sequential, parallel or DC-aware |
| Starting cadence in the Apache docs | Every 1 to 3 weeks (every 5 days if you skip incremental) |
Incremental repair
| Aspect | Incremental |
|---|---|
| Command | nodetool repair (the default) |
| Data compared | Only unrepaired SSTables |
| Anticompaction | At session start, to split repaired from unrepaired data |
| Marks data as repaired | Yes, sets repairedAt when the session finalizes |
Snapshots with -seq or -dcpar | No; the pending repair pool isolates the data |
| Catches corruption in already repaired data | No |
| Reaper parallelism | Parallel only |
| Starting cadence in the Apache docs | Every 1 to 3 days |
Run both: incremental repair never looks at repaired data again, so the Apache docs still call for occasional full repairs to catch disk corruption, operator error and bugs.
# Incremental (default) for one keyspace, primary ranges only
nodetool repair -pr shop
# Full repair of two tables
nodetool repair --full -pr shop orders order_itemsInterpretation: both commands repair only the node you run them on. Without table names, every table in the keyspace is repaired. To see how much data sits in each pool, read the per-table counters:
nodetool tablestats shop.orders | grep -iE 'repaired|pending repair'Interpretation: tablestats prints Percent repaired, Bytes repaired, Bytes unrepaired and Bytes pending repair. With incremental repair, a falling Percent repaired means the schedule is not keeping up.
The first incremental repair covers all existing data, so it costs a full repair plus anticompaction of every SSTable. The auto repair guide recommends a full repair cycle across the cluster first.
Check before you repair: preview and validate
Two flags build Merkle trees without streaming. --preview (-prv) estimates what a repair would stream. --validate (-vd) compares the repaired data across replicas.
nodetool repair -prv -pr shop # estimate an incremental repair
nodetool repair -prv --full -pr shop # estimate a full repair
nodetool repair -vd -pr shop # is repaired data in sync?Interpretation: a clean preview reports Previewed data was in sync; otherwise it prints Total estimated streaming: N ranges, N sstables, N bytes. If validate reports Repaired data is inconsistent, the docs say to run a full repair.
How do -pr and subrange repair split the work?
With replication factor 3, every range lives on three nodes. A plain nodetool repair on each node therefore repairs every range three times. -pr limits the run to the node's primary ranges, the ranges where it is the first replica. The 5.0 docs say to run nodetool repair -pr on each node in each datacenter to cover the cluster. Primary ranges are counted across the whole ring, so skipping a datacenter leaves its primary ranges unrepaired.
Cassandra rejects -pr combined with -dc or -hosts, with the error "Primary range repair should be performed on all nodes in the cluster." -pr -local is allowed.
Subrange repair goes one step further. You pass a start token (exclusive) and an end token (inclusive), so each session covers a small slice.
# List the token ranges and their replicas
nodetool describering shop
# Full repair of one slice, two tables at a time
nodetool repair --full -st -9223372036854775808 -et -9150000000000000000 \
-j 2 shopInterpretation: both tokens must fall inside one range that this node replicates. Smaller slices give finer Merkle trees for the same repair_session_space, so less over-streaming. A failed slice can be retried alone. -j sets how many tables are repaired at once (default 1, maximum 4).
A full primary-range repair of one keyspace
Illustrative output shaped like nodetool on Cassandra 5.0; IDs, tokens and timings are invented
Read the options line: confirm primary range: true and the incremental value before you walk away. The message formats come from the 5.0 repair coordinator source.
Sequential, parallel or DC-parallel: which repair mode?
Parallelism decides how many replicas build Merkle trees at the same time. In 4.1 and 5.0, nodetool repair defaults to parallel.
| Flag | Mode | Validation runs on | Snapshots (full repair) |
|---|---|---|---|
| (none) | Parallel | All replicas at once | No |
-seq | Sequential | One replica at a time | Yes, validation reads a snapshot |
-dcpar | DC-parallel | One replica per datacenter at a time | Yes |
# Keep two replicas of every range free to serve reads
nodetool repair --full -pr -seq shop
# Multi-DC: one replica per datacenter validates at a time
nodetool repair --full -pr -dcpar shopInterpretation: parallel repair is fastest but loads every replica of a range at once, which QUORUM reads can feel. Sequential and DC-parallel full repairs take snapshots, so check free disk space first.
How do you schedule repairs with Cassandra Reaper?
Reaper is an open source repair scheduler with a web UI, a REST API and the spreaper CLI. It talks to Cassandra over JMX. It is developed in the thelastpickle/cassandra-reaper repository on GitHub; The Last Pickle, which forked it from Spotify's original, joined DataStax in 2020. According to the release list, 5.1.0 (26 October 2026) is the current release, and 5.0.0 (8 July 2026) moved Reaper to Java 17 and Dropwizard 5. The 4.2 line still received patch releases in August 2026.
Reaper splits each repair run into segments, which are small subranges, and repairs them one at a time with back-pressure:
- Segments:
segmentCountPerNodedefaults to 64. A segment that runs longer than 30 minutes (hangingRepairTimeoutMins) is cancelled and retried later. - Intensity: a value from 0.0 to 1.0, never 0. At 0.5, Reaper spends as long sleeping between segments as repairing; at 1.0 it does not sleep.
- Back-pressure: Reaper postpones a segment if a replica is already repairing or has more than
maxPendingCompactions(default 20) pending compactions. - Incremental: supported only with
PARALLELparallelism.subrangeIncrementalRepairruns it per segment. - TWCS tables: the docs recommend
blacklistTwcsTables: true, because repairing TTL data in TWCS tables stops old SSTables from expiring.
# cassandra-reaper.yaml (values from the sample config in the repository)
segmentCountPerNode: 64
repairParallelism: DATACENTER_AWARE
repairIntensity: 0.9
scheduleDaysBetween: 7
hangingRepairTimeoutMins: 30
incrementalRepair: false
blacklistTwcsTables: true
maxParallelRepairs: 2
storageType: memory # use cassandra when more than one Reaper instance runs
datacenterAvailability: ALL # SIDECAR if JMX only listens on localhostInterpretation: the local memory backend (stored in SQLite since Reaper 4.2.0) supports a single Reaper instance; several instances need the Cassandra backend. Cassandra restricts JMX to localhost by default, and then only SIDECAR mode, one Reaper per node, can reach it.
Set up a Reaper repair schedule inside gc_grace_seconds
Six steps; step 4 gives the measured run time that the schedule in step 5 has to fit
-
Find the repair deadline
The smallest
gc_grace_secondsamong the tables you will schedule is the deadline. The default is 10 days.SELECT table_name, gc_grace_seconds FROM system_schema.tables WHERE keyspace_name = 'shop'; -
Log in and register the cluster
Give one seed node and its JMX port. Reaper discovers the other nodes.
spreaper login admin spreaper add-cluster cass-01.example.internal 7199 -
Choose segments, parallelism and intensity
Start with the defaults. Incremental schedules need PARALLEL.
-
Run one repair by hand and time it
The measured duration tells you how much of the grace period one run uses.
spreaper repair prod shop \ --tables orders,order_items \ --repair-parallelism DATACENTER_AWARE \ --intensity 0.9 -
Create the schedule
Days between runs plus run time must stay below the deadline: with 10 days of grace, 7 days plus a run under 3 days fits.
spreaper schedule prod shop \ --tables orders,order_items \ --schedule-days-between 7 \ --schedule-trigger-time 2026-11-15T01:00:00 -
Alert when the schedule falls behind
Watch run status with
spreaper list-runs. Since 4.2.0, Reaper exposes anunfulfilledRepairSchedulemetric for schedules that miss their window.
Main risk: a run that grows longer than the gap you left. Re-measure as data grows and shorten the interval in time.
The scheduling guide recommends schedules per group of tables rather than whole keyspaces. Running Reaper on Kubernetes through K8ssandra is a separate setup, not covered here.
Does Cassandra have built-in automatic repair now?
Yes, from 5.0.8. The 5.0 NEWS file lists CEP-37 Auto Repair under 5.0.8, which was announced on 16 April 2026. The scheduler runs inside Cassandra and keeps its history in system_distributed.auto_repair_history. It can run full, incremental and preview_repaired repairs, with token-range splitting and retries. The docs say it was built for Cassandra 6.0 and backported. As of November 2026, the download page lists no 6.0 release, and Cassandra 4.1 does not have the feature.
Turning it on takes two steps, and the first one is permanent:
# 1. jvm-server.options (or JVM_OPTS) on EVERY node, before startup.
# Creates schema elements; cannot be reverted once set.
-Dcassandra.autorepair.enable=true# 2. cassandra.yaml: shortened from the example in the 5.0 auto repair docs
auto_repair:
enabled: true
repair_type_overrides:
full:
enabled: true
min_repair_interval: 5d
incremental:
enabled: true
min_repair_interval: 1h
global_settings:
parallel_repair_count: 1Interpretation: the scheduler and each repair type are off by default, and min_repair_interval defaults to 24h. If only some nodes have the JVM flag, the system_distributed schema disagrees and schema changes time out until every node matches. A node that had the flag fails to start without it, so stop scheduling through cassandra.yaml or JMX instead.
nodetool getautorepairconfig
nodetool autorepairstatus -t incremental
nodetool setautorepairconfig -t full min_repair_interval 5d # this node only, lost on restartInterpretation: per-table control uses the new auto_repair table option, such as WITH auto_repair = {'incremental_enabled': 'false'}. Pick one scheduler per table: auto repair and Reaper on the same tables would overlap, and incremental sessions cannot share SSTables.
How do you monitor repair and spot when it falls behind?
-- cqlsh, Cassandra 4.1 and 5.0: repairs coordinated by this node
SELECT id, keyspace_name, status, completed, duration_millis, failure_cause
FROM system_views.repairs;Interpretation: repairs and its sibling tables (repair_sessions, repair_jobs, repair_validations) are node-local and held in memory, so query each coordinator and expect no history after a restart.
nodetool compactionstats # Validation and Anticompaction tasks in flight
nodetool netstats # repair streams and bytes remaining
nodetool repair_admin list # incremental sessions on this node
nodetool repair_admin summarize-pending shopInterpretation: a Validation row in compactionstats is the Merkle tree build, and it shares the compaction_throughput limit. A growing pending compaction count during repair is your cue to lower Reaper intensity or, on 5.0, set reject_repair_compaction_threshold.
For dashboards, the metrics reference lists per-table PercentRepaired and BytesUnrepaired gauges and a RepairTime timer. Alert on the age of the last successful repair per table.
What should you do when a repair fails?
Read the first error, not the last line. These are the message formats from the 5.0 source:
Repair session <id> for range <range> failed with error <message>
Repair command #<n> finished with error
Repair job has failed with the error message: <message>. Check the logs on the repair participants for further details
Cannot proceed on repair because a neighbor (<endpoint>) is dead: session failedInterpretation: the coordinator only reports that a session failed. The cause (a validation timeout, heap exhaustion during tree building, a broken stream) is in system.log on the failing participant, so search every replica's log for the session ID. Then check the usual causes:
- A replica is down. Repair needs every replica of the range.
--forceskips dead nodes, but a forced incremental repair then marks nothing as repaired. Bring the node back and rerun. - A stuck incremental session. List and cancel it, then clean up released data.bash
nodetool repair_admin list nodetool repair_admin cancel --session <session-id> nodetool repair_admin cleanup - Heap pressure while building trees. Lower
repair_session_space, setconcurrent_merkle_tree_requeststo 1, or repair smaller subranges. - Disk or compaction backlog. On 5.0,
repair_disk_headroom_reject_ratio(disabled by default) andreject_repair_compaction_thresholdmake Cassandra refuse new repairs instead. - Retry small. Rerun only the failed ranges with
-stand-et. Add-trto trace the run intosystem_traces.events.
If the deadline is close, raising gc_grace_seconds on the affected tables buys time, at the cost of more tombstones on disk. Set it back once repairs are healthy.
Frequently asked questions
How often should I run nodetool repair in Cassandra?
Often enough that every node finishes a repair within gc_grace_seconds, which defaults to 10 days. The Apache Cassandra docs suggest incremental repair every 1 to 3 days and full repair every 1 to 3 weeks, or a full repair every 5 days without incremental. With the default grace period, repair every node at least every 7 days.
Is incremental repair safe to use in Cassandra 4.1 and 5.0?
It is the default, and the 4.0 redesign fixed the over-streaming problems of 3.x incremental repair. SSTables are now anticompacted before Merkle trees are built and marked repaired only when the session succeeds. Run a full repair cycle before your first incremental repair, and keep running occasional full repairs to catch corruption in data already marked repaired.
What happens if a Cassandra node is down longer than gc_grace_seconds?
Other replicas may have purged the tombstones the node missed. If you start it and run repair, its stale copies of deleted rows can be copied back to the cluster as zombie data. The usual remedy is to wipe the node and bring it back as a replacement node, so it streams current data from healthy replicas.
Does Cassandra 5.0 have automatic repair, or do I still need Reaper?
Cassandra 5.0.8 and later include the CEP-37 auto repair scheduler. It is off by default and needs the JVM property cassandra.autorepair.enable=true on every node, which cannot be reverted. Cassandra 4.1 has no built-in scheduler, so Reaper or a cron script is still needed there. Use one scheduler per table, not both.
Key takeaways
- The repair deadline is the smallest
gc_grace_secondsyou use: every node must complete a repair inside it, or deleted data can return. - Use incremental repair (the default) for frequent runs and full repair every 1 to 3 weeks. Run a full cycle before switching an existing cluster to incremental.
- Cover the ring once:
-pron every node in every datacenter, or subranges with-stand-et. Choose-seqor-dcparwhen parallel validation hurts read latency. - Schedule with Reaper 5.1.0 (segments, intensity, back-pressure) or, on 5.0.8 and later, the built-in auto repair, whose enabling flag is permanent. Pick one per table.
- Alert on the age of the last successful repair and on
PercentRepaired. When a repair fails, read the participant logs, clear stuck sessions withrepair_admin, and retry small ranges.
Get your Cassandra repair schedule reviewed with JusDB
JusDB's Cassandra support team can review your repair history against each table's gc_grace_seconds, plan a move from manual scripts to Reaper or to the 5.0 auto repair scheduler, and help run the first full and incremental cycles on large clusters.
Bring your versions, topology, current repair scripts or Reaper configuration, and recent repair errors. Your team approves every production change.