JusDB
Free DB Audit
NoSQL Databases

Cassandra nodetool repair: Full vs Incremental, and Scheduling with Reaper

Run Cassandra 4.1 and 5.0 repairs that finish inside gc_grace_seconds: full vs incremental, -pr and subranges, parallelism, Reaper schedules, the built-in auto repair, and failed repairs.

JusDB Team
Published October 30, 2026
Updated November 9, 2026
18 min read

A row you deleted last month is back. Or a repair that used to take an hour now runs for two days and ends with Repair command #12 finished with error. On Apache Cassandra 4.1 and 5.0, both problems come from the same place: anti-entropy repair is not running often enough, or not finishing. Cassandra does not repair data on a schedule unless you configure it. Hints cover short outages only. Any replica that misses a delete and is not repaired within gc_grace_seconds can bring the deleted data back. This guide covers full and incremental repair, -pr, subranges and parallelism, scheduling with Cassandra Reaper or the auto repair scheduler in Cassandra 5.0.8, and failed repairs.

TL;DR
  • Every node must finish a repair within gc_grace_seconds (default 864000, which is 10 days). Otherwise deleted data can come back. The Apache docs suggest repairing every node at least every 7 days.
  • Incremental repair is the default on 4.1 and 5.0 and only compares unrepaired SSTables. Since 4.0 it isolates its SSTables before it builds Merkle trees, which fixed the over-streaming problems of 3.x. Still run a full repair (--full) every 1 to 3 weeks.
  • Run nodetool repair -pr on every node in every datacenter, or split the ring into subranges with -st and -et.
  • Reaper 5.1.0 (26 October 2026) schedules segmented repairs with back-pressure. Cassandra 5.0.8 and later also include the CEP-37 auto repair scheduler. It is off by default, and the JVM flag that enables it cannot be turned off again.

Scope and review date: November 2026. This post covers Apache Cassandra 4.1 (latest 4.1.12) and 5.0 (latest 5.0.9), and Reaper 5.1.0. Commands, defaults and messages were checked on 9 November 2026 against the Apache Cassandra documentation, the 4.1 and 5.0 source branches and the Reaper documentation. They were not run in a lab, so rehearse them on a staging cluster first.


Why does Cassandra need repair at all?

Cassandra keeps writing when a replica is down, so replicas drift apart. Hints replay missed writes only for outages shorter than max_hint_window (3 hours by default), and the repair documentation calls them best effort. Anti-entropy repair compares replicas and copies the differences.

Deletes make this urgent. A delete writes a tombstone, which compaction may drop once it is older than gc_grace_seconds. A replica that missed the tombstone still holds the old value, and once the tombstone is gone elsewhere, repair copies that value back. The tombstones page calls this deleted data a zombie.

sql
-- cqlsh: the grace period for every table in one keyspace
SELECT table_name, gc_grace_seconds
FROM system_schema.tables
WHERE keyspace_name = 'shop';

Interpretation: 864000 seconds is the 10-day default. The shortest value in the keyspace sets your repair deadline. Every node must complete a repair of that table in less time than that, with slack for failed runs.

A node that was down longer than gc_grace_seconds

Do not just start it and run repair. The tombstones it missed may already be purged elsewhere, so its old data would be repaired back into the cluster. The usual fix is to wipe the node and bring it back as a replacement, so it streams current data from the other replicas.

What does nodetool repair actually do?

nodetool repair repairs only the token ranges of the node it runs on. For each range and table, that node (the coordinator) asks every replica for a Merkle tree, a tree of hashes built by a validation compaction that reads every SSTable in the range. It compares the trees and streams only the ranges whose hashes differ.

One repair session, from Merkle trees to streaming

The validation read in step 2 happens on every replica, even when nothing is out of sync

How nodetool repair compares and syncs replicas Step 1: the coordinator node, where nodetool repair runs, starts a session for a token range. Step 2: every replica runs a validation compaction that reads its SSTables and builds a Merkle tree. Step 3: the coordinator compares the trees. Step 4: replicas stream only the ranges whose hashes differ, and the streamed data is written as new SSTables. Coordinator nodetool repair Validation Merkle tree per replica Compare diff the hashes Stream mismatched ranges 1 2 3 4 reads every SSTable in range coarse trees over-stream writes new SSTables Tree size per table and range is capped by repair_session_space (default 1/16 of heap)

Where the cost is: validation is a full read of the range on each replica, and streaming adds writes and compaction. Incremental repair cuts the validation input to unrepaired SSTables only.

Four cassandra.yaml settings, the same in 4.1 and 5.0, bound that cost:

yaml
# cassandra.yaml (5.0 defaults shown as comments in the shipped file)
# repair_session_space:            # default 1/16 of heap, minimum 1 MiB
# concurrent_merkle_tree_requests: 0   # 0 = no limit
# concurrent_validations: 0            # below 1 = same as concurrent_compactors
compaction_throughput: 64MiB/s         # also throttles validation compactions

Interpretation: with a smaller repair_session_space, each Merkle tree leaf covers more data, so one changed row streams a bigger range. The shipped file says to raise it, or use subrange repair, against over-streaming, and to lower it under heap pressure.

Full or incremental repair: which should you run?

Both 4.1 and 5.0 run incremental repair when you type nodetool repair, and full repair with --full. Incremental repair has a bad history: before 4.0, SSTables could be compacted between Merkle tree building and anticompaction. They were then never marked repaired, and the next repair over-streamed them (CASSANDRA-9143). The Last Pickle's write-up of the 4.0 change describes the fix. Cassandra now anticompacts first and moves the candidate SSTables into a pending repair pool that only that session uses. It sets their repairedAt timestamp only when the whole session succeeds. Full repair no longer anticompacts and no longer marks anything repaired.

Full and incremental repair on Cassandra 4.1 and 5.0

Switch types to see what each one reads, rewrites and marks

Full repair

AspectFull
Commandnodetool repair --full
Data comparedAll SSTables in the range
AnticompactionNone since 4.0
Marks data as repairedNo
Snapshots with -seq or -dcparYes, on every replica
Catches corruption in already repaired dataYes
Reaper parallelismSequential, parallel or DC-aware
Starting cadence in the Apache docsEvery 1 to 3 weeks (every 5 days if you skip incremental)

Incremental repair

AspectIncremental
Commandnodetool repair (the default)
Data comparedOnly unrepaired SSTables
AnticompactionAt session start, to split repaired from unrepaired data
Marks data as repairedYes, sets repairedAt when the session finalizes
Snapshots with -seq or -dcparNo; the pending repair pool isolates the data
Catches corruption in already repaired dataNo
Reaper parallelismParallel only
Starting cadence in the Apache docsEvery 1 to 3 days

Run both: incremental repair never looks at repaired data again, so the Apache docs still call for occasional full repairs to catch disk corruption, operator error and bugs.

bash
# Incremental (default) for one keyspace, primary ranges only
nodetool repair -pr shop

# Full repair of two tables
nodetool repair --full -pr shop orders order_items

Interpretation: both commands repair only the node you run them on. Without table names, every table in the keyspace is repaired. To see how much data sits in each pool, read the per-table counters:

bash
nodetool tablestats shop.orders | grep -iE 'repaired|pending repair'

Interpretation: tablestats prints Percent repaired, Bytes repaired, Bytes unrepaired and Bytes pending repair. With incremental repair, a falling Percent repaired means the schedule is not keeping up.

Switching an existing cluster to incremental

The first incremental repair covers all existing data, so it costs a full repair plus anticompaction of every SSTable. The auto repair guide recommends a full repair cycle across the cluster first.

Check before you repair: preview and validate

Two flags build Merkle trees without streaming. --preview (-prv) estimates what a repair would stream. --validate (-vd) compares the repaired data across replicas.

bash
nodetool repair -prv -pr shop          # estimate an incremental repair
nodetool repair -prv --full -pr shop   # estimate a full repair
nodetool repair -vd -pr shop           # is repaired data in sync?

Interpretation: a clean preview reports Previewed data was in sync; otherwise it prints Total estimated streaming: N ranges, N sstables, N bytes. If validate reports Repaired data is inconsistent, the docs say to run a full repair.

How do -pr and subrange repair split the work?

With replication factor 3, every range lives on three nodes. A plain nodetool repair on each node therefore repairs every range three times. -pr limits the run to the node's primary ranges, the ranges where it is the first replica. The 5.0 docs say to run nodetool repair -pr on each node in each datacenter to cover the cluster. Primary ranges are counted across the whole ring, so skipping a datacenter leaves its primary ranges unrepaired.

Cassandra rejects -pr combined with -dc or -hosts, with the error "Primary range repair should be performed on all nodes in the cluster." -pr -local is allowed.

Subrange repair goes one step further. You pass a start token (exclusive) and an end token (inclusive), so each session covers a small slice.

bash
# List the token ranges and their replicas
nodetool describering shop

# Full repair of one slice, two tables at a time
nodetool repair --full -st -9223372036854775808 -et -9150000000000000000 \
  -j 2 shop

Interpretation: both tokens must fall inside one range that this node replicates. Smaller slices give finer Merkle trees for the same repair_session_space, so less over-streaming. A failed slice can be retried alone. -j sets how many tables are repaired at once (default 1, maximum 4).

A full primary-range repair of one keyspace

Illustrative output shaped like nodetool on Cassandra 5.0; IDs, tokens and timings are invented

cassandra@cass-01:~
$ nodetool repair --full -pr shop[2026-10-29 01:00:02,118] Starting repair command #12 (c2a4e3f0-...), repairing keyspace shop with repair options (parallelism: parallel, primary range: true, incremental: false, job threads: 1, ...)[2026-10-29 01:03:47,560] Repair session 0f1b77a0-... for range [(-918734323983581184,-915131444281684787]] finished (progress: 6%)...[2026-10-29 01:41:15,902] Repair session 7d55c210-... for range [(908421338700271411,912024218402204262]] finished (progress: 100%)[2026-10-29 01:41:16,004] Repair completed successfully[2026-10-29 01:41:16,011] Repair command #12 finished in 41 minutes 13 seconds

Read the options line: confirm primary range: true and the incremental value before you walk away. The message formats come from the 5.0 repair coordinator source.

Sequential, parallel or DC-parallel: which repair mode?

Parallelism decides how many replicas build Merkle trees at the same time. In 4.1 and 5.0, nodetool repair defaults to parallel.

FlagModeValidation runs onSnapshots (full repair)
(none)ParallelAll replicas at onceNo
-seqSequentialOne replica at a timeYes, validation reads a snapshot
-dcparDC-parallelOne replica per datacenter at a timeYes
bash
# Keep two replicas of every range free to serve reads
nodetool repair --full -pr -seq shop

# Multi-DC: one replica per datacenter validates at a time
nodetool repair --full -pr -dcpar shop

Interpretation: parallel repair is fastest but loads every replica of a range at once, which QUORUM reads can feel. Sequential and DC-parallel full repairs take snapshots, so check free disk space first.

How do you schedule repairs with Cassandra Reaper?

Reaper is an open source repair scheduler with a web UI, a REST API and the spreaper CLI. It talks to Cassandra over JMX. It is developed in the thelastpickle/cassandra-reaper repository on GitHub; The Last Pickle, which forked it from Spotify's original, joined DataStax in 2020. According to the release list, 5.1.0 (26 October 2026) is the current release, and 5.0.0 (8 July 2026) moved Reaper to Java 17 and Dropwizard 5. The 4.2 line still received patch releases in August 2026.

Reaper splits each repair run into segments, which are small subranges, and repairs them one at a time with back-pressure:

  • Segments: segmentCountPerNode defaults to 64. A segment that runs longer than 30 minutes (hangingRepairTimeoutMins) is cancelled and retried later.
  • Intensity: a value from 0.0 to 1.0, never 0. At 0.5, Reaper spends as long sleeping between segments as repairing; at 1.0 it does not sleep.
  • Back-pressure: Reaper postpones a segment if a replica is already repairing or has more than maxPendingCompactions (default 20) pending compactions.
  • Incremental: supported only with PARALLEL parallelism. subrangeIncrementalRepair runs it per segment.
  • TWCS tables: the docs recommend blacklistTwcsTables: true, because repairing TTL data in TWCS tables stops old SSTables from expiring.
yaml
# cassandra-reaper.yaml (values from the sample config in the repository)
segmentCountPerNode: 64
repairParallelism: DATACENTER_AWARE
repairIntensity: 0.9
scheduleDaysBetween: 7
hangingRepairTimeoutMins: 30
incrementalRepair: false
blacklistTwcsTables: true
maxParallelRepairs: 2
storageType: memory        # use cassandra when more than one Reaper instance runs
datacenterAvailability: ALL   # SIDECAR if JMX only listens on localhost

Interpretation: the local memory backend (stored in SQLite since Reaper 4.2.0) supports a single Reaper instance; several instances need the Cassandra backend. Cassandra restricts JMX to localhost by default, and then only SIDECAR mode, one Reaper per node, can reach it.

Set up a Reaper repair schedule inside gc_grace_seconds

Six steps; step 4 gives the measured run time that the schedule in step 5 has to fit

  1. Find the repair deadline

    The smallest gc_grace_seconds among the tables you will schedule is the deadline. The default is 10 days.

    SELECT table_name, gc_grace_seconds FROM system_schema.tables WHERE keyspace_name = 'shop';
  2. Log in and register the cluster

    Give one seed node and its JMX port. Reaper discovers the other nodes.

    spreaper login admin spreaper add-cluster cass-01.example.internal 7199
  3. Choose segments, parallelism and intensity

    Start with the defaults. Incremental schedules need PARALLEL.

  4. Run one repair by hand and time it

    The measured duration tells you how much of the grace period one run uses.

    spreaper repair prod shop \ --tables orders,order_items \ --repair-parallelism DATACENTER_AWARE \ --intensity 0.9
  5. Create the schedule

    Days between runs plus run time must stay below the deadline: with 10 days of grace, 7 days plus a run under 3 days fits.

    spreaper schedule prod shop \ --tables orders,order_items \ --schedule-days-between 7 \ --schedule-trigger-time 2026-11-15T01:00:00
  6. Alert when the schedule falls behind

    Watch run status with spreaper list-runs. Since 4.2.0, Reaper exposes an unfulfilledRepairSchedule metric for schedules that miss their window.

Main risk: a run that grows longer than the gap you left. Re-measure as data grows and shorten the interval in time.

The scheduling guide recommends schedules per group of tables rather than whole keyspaces. Running Reaper on Kubernetes through K8ssandra is a separate setup, not covered here.

Does Cassandra have built-in automatic repair now?

Yes, from 5.0.8. The 5.0 NEWS file lists CEP-37 Auto Repair under 5.0.8, which was announced on 16 April 2026. The scheduler runs inside Cassandra and keeps its history in system_distributed.auto_repair_history. It can run full, incremental and preview_repaired repairs, with token-range splitting and retries. The docs say it was built for Cassandra 6.0 and backported. As of November 2026, the download page lists no 6.0 release, and Cassandra 4.1 does not have the feature.

Turning it on takes two steps, and the first one is permanent:

bash
# 1. jvm-server.options (or JVM_OPTS) on EVERY node, before startup.
#    Creates schema elements; cannot be reverted once set.
-Dcassandra.autorepair.enable=true
yaml
# 2. cassandra.yaml: shortened from the example in the 5.0 auto repair docs
auto_repair:
  enabled: true
  repair_type_overrides:
    full:
      enabled: true
      min_repair_interval: 5d
    incremental:
      enabled: true
      min_repair_interval: 1h
  global_settings:
    parallel_repair_count: 1

Interpretation: the scheduler and each repair type are off by default, and min_repair_interval defaults to 24h. If only some nodes have the JVM flag, the system_distributed schema disagrees and schema changes time out until every node matches. A node that had the flag fails to start without it, so stop scheduling through cassandra.yaml or JMX instead.

bash
nodetool getautorepairconfig
nodetool autorepairstatus -t incremental
nodetool setautorepairconfig -t full min_repair_interval 5d   # this node only, lost on restart

Interpretation: per-table control uses the new auto_repair table option, such as WITH auto_repair = {'incremental_enabled': 'false'}. Pick one scheduler per table: auto repair and Reaper on the same tables would overlap, and incremental sessions cannot share SSTables.

How do you monitor repair and spot when it falls behind?

sql
-- cqlsh, Cassandra 4.1 and 5.0: repairs coordinated by this node
SELECT id, keyspace_name, status, completed, duration_millis, failure_cause
FROM system_views.repairs;

Interpretation: repairs and its sibling tables (repair_sessions, repair_jobs, repair_validations) are node-local and held in memory, so query each coordinator and expect no history after a restart.

bash
nodetool compactionstats      # Validation and Anticompaction tasks in flight
nodetool netstats             # repair streams and bytes remaining
nodetool repair_admin list    # incremental sessions on this node
nodetool repair_admin summarize-pending shop

Interpretation: a Validation row in compactionstats is the Merkle tree build, and it shares the compaction_throughput limit. A growing pending compaction count during repair is your cue to lower Reaper intensity or, on 5.0, set reject_repair_compaction_threshold.

For dashboards, the metrics reference lists per-table PercentRepaired and BytesUnrepaired gauges and a RepairTime timer. Alert on the age of the last successful repair per table.

What should you do when a repair fails?

Read the first error, not the last line. These are the message formats from the 5.0 source:

text
Repair session <id> for range <range> failed with error <message>
Repair command #<n> finished with error
Repair job has failed with the error message: <message>. Check the logs on the repair participants for further details
Cannot proceed on repair because a neighbor (<endpoint>) is dead: session failed

Interpretation: the coordinator only reports that a session failed. The cause (a validation timeout, heap exhaustion during tree building, a broken stream) is in system.log on the failing participant, so search every replica's log for the session ID. Then check the usual causes:

  1. A replica is down. Repair needs every replica of the range. --force skips dead nodes, but a forced incremental repair then marks nothing as repaired. Bring the node back and rerun.
  2. A stuck incremental session. List and cancel it, then clean up released data.
    bash
    nodetool repair_admin list
    nodetool repair_admin cancel --session <session-id>
    nodetool repair_admin cleanup
  3. Heap pressure while building trees. Lower repair_session_space, set concurrent_merkle_tree_requests to 1, or repair smaller subranges.
  4. Disk or compaction backlog. On 5.0, repair_disk_headroom_reject_ratio (disabled by default) and reject_repair_compaction_threshold make Cassandra refuse new repairs instead.
  5. Retry small. Rerun only the failed ranges with -st and -et. Add -tr to trace the run into system_traces.events.

If the deadline is close, raising gc_grace_seconds on the affected tables buys time, at the cost of more tombstones on disk. Set it back once repairs are healthy.

Frequently asked questions

How often should I run nodetool repair in Cassandra?

Often enough that every node finishes a repair within gc_grace_seconds, which defaults to 10 days. The Apache Cassandra docs suggest incremental repair every 1 to 3 days and full repair every 1 to 3 weeks, or a full repair every 5 days without incremental. With the default grace period, repair every node at least every 7 days.

Is incremental repair safe to use in Cassandra 4.1 and 5.0?

It is the default, and the 4.0 redesign fixed the over-streaming problems of 3.x incremental repair. SSTables are now anticompacted before Merkle trees are built and marked repaired only when the session succeeds. Run a full repair cycle before your first incremental repair, and keep running occasional full repairs to catch corruption in data already marked repaired.

What happens if a Cassandra node is down longer than gc_grace_seconds?

Other replicas may have purged the tombstones the node missed. If you start it and run repair, its stale copies of deleted rows can be copied back to the cluster as zombie data. The usual remedy is to wipe the node and bring it back as a replacement node, so it streams current data from healthy replicas.

Does Cassandra 5.0 have automatic repair, or do I still need Reaper?

Cassandra 5.0.8 and later include the CEP-37 auto repair scheduler. It is off by default and needs the JVM property cassandra.autorepair.enable=true on every node, which cannot be reverted. Cassandra 4.1 has no built-in scheduler, so Reaper or a cron script is still needed there. Use one scheduler per table, not both.

Key takeaways

  • The repair deadline is the smallest gc_grace_seconds you use: every node must complete a repair inside it, or deleted data can return.
  • Use incremental repair (the default) for frequent runs and full repair every 1 to 3 weeks. Run a full cycle before switching an existing cluster to incremental.
  • Cover the ring once: -pr on every node in every datacenter, or subranges with -st and -et. Choose -seq or -dcpar when parallel validation hurts read latency.
  • Schedule with Reaper 5.1.0 (segments, intensity, back-pressure) or, on 5.0.8 and later, the built-in auto repair, whose enabling flag is permanent. Pick one per table.
  • Alert on the age of the last successful repair and on PercentRepaired. When a repair fails, read the participant logs, clear stuck sessions with repair_admin, and retry small ranges.

Get your Cassandra repair schedule reviewed with JusDB

JusDB's Cassandra support team can review your repair history against each table's gc_grace_seconds, plan a move from manual scripts to Reaper or to the 5.0 auto repair scheduler, and help run the first full and incremental cycles on large clusters.

Bring your versions, topology, current repair scripts or Reaper configuration, and recent repair errors. Your team approves every production change.

Share this article

Database engineering notes

Articles like this one, in your inbox. No spam, unsubscribe anytime.

JusDB Team

Official JusDB content team

Keep reading

Moving a Multi-Database Valkey Instance into a 9.x Cluster

Move a Valkey instance that uses SELECT 1 to 15 into a 9.x cluster: keep or remap databases, copy each one, verify counts per database, and keep a way back.

Redis18 minNov 10, 2026
Read

Valkey Keyspace Notifications in Production: Expired Events, hexpired and Cluster Mode

Run Valkey keyspace notifications in production: why expired and hexpired events lag the TTL, how to subscribe on every cluster node, and when a stream is safer.

Redis20 minNov 10, 2026
Read

Valkey GLIDE in Production: AZ-Affinity Reads and Migrating from Legacy Clients

Configure Valkey GLIDE AZ-affinity reads on servers and clients, learn what GLIDE 2.5.3 does not support, and migrate from redis-py, Jedis, ioredis, or go-redis one service at a time.

Redis20 minNov 10, 2026
Read