MongoDB Change Streams in Production: Resume Tokens and the Oplog Window

A change stream consumer that has run for months stops during a long outage, and the restart fails with Command failed with error 286 (ChangeStreamHistoryLost). Its saved resume token points at an oplog entry that MongoDB has already deleted. Change streams are the supported way to read MongoDB's oplog for cache invalidation, search indexing, and change data capture (CDC). On MongoDB 8.0 and 9.0, running them in production comes down to where the resume token is stored, how much oplog history stands between the consumer and that error, and how large events get. This guide covers those, plus pre- and post-images, sharded clusters, and the 9.0 change after elections.

Scope and review date: November 2026. This post covers change streams on self-managed MongoDB 8.0 and 9.0 replica sets and sharded clusters, with notes for MongoDB Atlas. Options, defaults, error codes, and version numbers were checked against MongoDB's documentation on 9 November 2026. The commands were not run in a lab, so rehearse them on staging.

How does a MongoDB change stream read the oplog?

A change stream is a cursor that returns one event per change, opened on a collection, a database, or a whole deployment. MongoDB builds each event from the oplog and only returns changes written to a majority of data-bearing members, so a rollback cannot take back an event you have processed.

The change streams documentation requires a replica set or sharded cluster, WiredTiger, and replica set protocol version 1. Open the stream on any data-bearing member of a replica set, or through mongos on a sharded cluster. System collections, the admin, local, and config databases, and time series collections cannot be watched.

// mongosh: one collection
const c1 = db.getSiblingDB("shop").orders.watch()

// One database: every non-system collection in "shop"
const c2 = db.getSiblingDB("shop").watch()

// The whole deployment: every database except admin, local, and config
const c3 = db.getMongo().watch()

Interpretation: all three return the same event format. A wider scope means fewer cursors, but more events to filter with $match.

With access control enabled, the consumer needs the find and changeStream actions on what it watches.

db.getSiblingDB("admin").createRole( {
   role: "ordersChangeStream",
   privileges: [
      { resource: { db: "shop", collection: "orders" },
        actions: [ "find", "changeStream" ] }
   ],
   roles: []
} )

Interpretation: for a database-level stream, set collection: ""; for a deployment-level stream, set both fields to "".

Which should you use: resumeAfter, startAfter, or startAtOperationTime?

Every change event carries its resume token in the _id field. The documentation describes two kinds: an event token, attached to a real change, and a highwatermark token, which the server generates periodically to show that cluster time has moved on with no matching events. Pass the entire _id document back, never a field inside it. The event reference says the fields inside _id depend on the MongoDB version and sometimes the feature compatibility version, so treat the token as opaque.

The three options are mutually exclusive, and each one fails if its starting point is no longer in the oplog. The db.collection.watch() reference adds one more rule in its resume section: reopen with the same pipeline and options that produced the token. A different pipeline can stop the stream from resuming or affect data consistency.

You do not have to wait for an event to get a token. The aggregate and getMore responses carry cursor.postBatchResumeToken, even when the batch is empty, and drivers expose it: PyMongo's ChangeStream.resume_token returns the token for the most recent change or batch. For debugging, the mongosh resumetoken snippet adds decodeResumeToken().

How do you store resume tokens so a consumer can restart safely?

Drivers make one automatic resume attempt after certain errors, using the token in memory. A restart or crash loses it, so the consumer saves its own: apply the change, save the token, and make the apply safe to repeat. A crash between the two replays one event, so delivery after a restart is at least once.

Do not deduplicate on clusterTime. Since MongoDB 8.0, a bulk insert can share one oplog entry and one clusterTime, and events with the same clusterTime can come from different transactions. Key target writes on documentKey.

# PyMongo 4.x: collection-level consumer with a durable checkpoint
from datetime import datetime, timezone
from pymongo import MongoClient

client = MongoClient(SOURCE_URI)
orders = client.shop.orders
target = MongoClient(TARGET_URI).reporting.orders
ckpt = client.ops.cs_checkpoints

PIPELINE = [{"$match": {"operationType": {"$in": ["insert", "update", "replace", "delete"]}}}]

def key_filter(event):
    return {k: {"$eq": v} for k, v in event["documentKey"].items()}

def apply(event):
    op = event["operationType"]
    if op == "delete":
        target.delete_one(key_filter(event))
    elif event.get("fullDocument") is not None:
        target.replace_one(key_filter(event), event["fullDocument"], upsert=True)

saved = ckpt.find_one({"_id": "orders-sync"})
resume = {"resume_after": saved["token"]} if saved else {}

with orders.watch(PIPELINE, full_document="updateLookup", **resume) as stream:
    while stream.alive:
        event = stream.try_next()
        if event is not None:
            apply(event)
        token = stream.resume_token        # event token or post-batch token
        if token is not None:
            ckpt.update_one({"_id": "orders-sync"},
                            {"$set": {"token": token,
                                      "savedAt": datetime.now(timezone.utc)}},
                            upsert=True)

Interpretation: the token is saved after every apply and on empty polls, so a quiet collection's checkpoint keeps moving. Replays rewrite the same document and deletes replay as no-ops, so the target converges. On a busy stream, save every few seconds instead; the cost is a longer replay after a crash.

When the target is in the same MongoDB deployment, you can write the change and the token in one transaction, so a restart resumes exactly after the last applied event.

What happens when a consumer falls behind the oplog window?

The oplog is a capped collection. On WiredTiger, its default size is 5% of free disk space, at least 990 MB and at most 50 GB, and by default MongoDB sets no minimum retention time (replica set oplog). The oplog window is the time between its oldest and newest entries. Once the entry behind a saved token is truncated, the stream cannot resume, and the error is not retryable. The MongoDB Kafka connector's troubleshooting page quotes it as:

Command failed with error 286 (ChangeStreamHistoryLost): 'PlanExecutor error
during aggregation :: caused by :: Resume of change stream was not possible,
as the resume point may no longer be in the oplog...

Interpretation: 286 is a non-resumable error. Retrying the same token fails the same way, so alert and stop instead of looping. Measure the window on every primary, including every shard's.

// mongosh, on each primary
rs.printReplicationInfo()

Interpretation: log length start to end is the current window. It shrinks during write bursts such as bulk loads, so read it at peak write volume. A mongos opens a change stream on each shard, so plan for the shortest window among them.

// How far is the saved checkpoint from the oldest oplog entry?
const oldest = new Date(db.getReplicationInfo().tFirst)
const saved = db.getSiblingDB("ops").cs_checkpoints.findOne({ _id: "orders-sync" })
const headroomHours = (saved.savedAt - oldest) / 3600000
headroomHours

Interpretation: savedAt approximates the token's position when the checkpoint moves on empty polls. A negative result means the token is gone. Alert well before zero.

If the window is shorter than the longest outage a consumer must survive, enlarge the oplog or set a minimum retention period on every data-bearing member. Our oplog sizing guide covers how to size it.

// Keep at least 48 hours of oplog on this member (example value)
db.adminCommand( { replSetResizeOplog: 1, minRetentionHours: 48 } )

Interpretation: the oplog then grows past its configured size to keep those hours. The replSetResizeOplog reference warns this can exhaust disk space under heavy writes. On Atlas, use the cluster's minimum oplog window setting.

How do you recover a consumer after downtime?

Most restarts are a plain resume. The runbook also covers the case where the token is gone.

If you use the Kafka connector or Debezium

Both read change streams and keep the resume position as Kafka Connect offsets, so the same oplog limit applies. The MongoDB Kafka connector (v3.1 docs) recovers from a lost token by moving to a new offset.partition.name. The Debezium MongoDB connector (3.7 docs) re-snapshots with snapshot.mode=when_needed, and its heartbeat.interval.ms defaults to 0 (off).

# MongoDB Kafka source connector (interval is an example value)
heartbeat.interval.ms=10000

# Debezium MongoDB connector
heartbeat.interval.ms=10000
snapshot.mode=when_needed
capture.mode=change_streams_update_full_with_pre_image

Interpretation: Debezium's default capture.mode is change_streams_update_full. The _with_pre_image modes need changeStreamPreAndPostImages on each captured collection. Our Debezium CDC guide covers the connector setup.

Should you use updateLookup or pre- and post-images?

By default, an update event carries only the delta in updateDescription. fullDocument: "updateLookup" reads the document again when the event is produced. Pre- and post-images, available since MongoDB 6.0, keep the document's state at the time of the change.

On sharded collections, updateLookup can also omit fullDocument when the update changed a shard key field. Images need one command per collection and one cluster parameter for retention.

// Enable images on one collection
db.runCommand( { collMod: "orders", changeStreamPreAndPostImages: { enabled: true } } )

// Keep pre-images for at most one day (example value; not supported on Atlas)
db.adminCommand( { setClusterParameter: {
   changeStreamOptions: { preAndPostImages: { expireAfterSeconds: 86400 } }
} } )

// Consume them
db.orders.watch( [], { fullDocument: "whenAvailable",
                       fullDocumentBeforeChange: "whenAvailable" } )

Interpretation: if expireAfterSeconds is off, images live as long as their oplog entries. Set retention longer than the consumer's longest expected outage, or after a resume whenAvailable returns events without images and required fails. Watch the size with db.getSiblingDB("config").system.preimages.totalSize() on each member.

How do you handle change events larger than 16 MB?

A change event is a BSON document with the same 16 MB limit, and an event over it closes the cursor with BSONObjectTooLarge. The documentation suggests documents under 8 MB if you request both images. Shrink events first with a $project stage and by requesting images only when needed. Otherwise, $changeStreamSplitLargeEvent (new in MongoDB 7.0 and 6.0.9) splits oversized events into fragments. It must be the last stage, only one is allowed, and it splits on top-level fields only: a single 20 MB field still fails.

const cur = db.orders.watch(
   [ { $changeStreamSplitLargeEvent: {} } ],
   { fullDocument: "required", fullDocumentBeforeChange: "required" }
)

let parts = []
while ( !cur.isClosed() ) {
   if ( !cur.hasNext() ) continue
   const ev = cur.next()
   if ( !ev.splitEvent ) { handle(ev); saveToken(ev._id); continue }
   parts.push(ev)
   if ( ev.splitEvent.fragment === ev.splitEvent.of ) {
      handle( Object.assign( {}, ...parts ) )   // merge top-level fields
      saveToken( ev._id )                       // token of the last fragment
      parts = []
   }
}

Interpretation: resuming from a fragment's token starts at the next fragment, so a saved middle token restarts halfway through an event. Save the token only after the final fragment. The stage is resumable only through watch(), not aggregate().

What changes on sharded clusters, and how much load do change streams add?

On a sharded cluster, mongos opens a change stream on every shard, whatever the shard key range you care about, then merges, sorts, and filters the results. It also runs the updateLookup reads. The production recommendations say the merge uses a global logical clock, so events can be processed in the order received.

For each event, mongos checks that every shard has moved past it, so a cold shard adds latency. More frequent no-op writes help: periodicNoopIntervalSecs defaults to 10 seconds, is set at startup, and on Atlas needs Atlas Support.

# mongod.conf on each shard member; takes effect at restart
setParameter:
  periodicNoopIntervalSecs: 1

Interpretation: 1 second is the documentation's example. If a busy sharded collection outruns mongos, filter with $match. Removing a shard can close a stream that may not be fully resumable; since MongoDB 8.0, movePrimary no longer invalidates streams.

On load: the oplog has no indexes, so every stream scans it, and many narrow streams repeat that scan. Each open stream also holds a connection in getMore, so keep maxPoolSize above the stream count. MongoDB 8.2.0 had a change stream regression of up to 15% against 8.0, fixed in 8.2.1.

// MongoDB 9.0: change stream counters on this mongod or mongos
const cs = db.serverStatus().metrics.changeStreams
cs.cursor.open.total                          // open change stream cursors now
cs.cursor.totalOpened                         // cumulative; includes resumes
cs.error.nonRetriable.changeStreamHistoryLost // failed resumes: token gone
cs.error.retriable.interruptedDueToReplStateChange

Interpretation: these counters are new in 9.0 (serverStatus reference). A fast-rising totalOpened means frequent reconnects; any changeStreamHistoryLost increase means a re-copy.

Which errors and events end a change stream?

Some interruptions resume on their own; others end the stream. Codes are from MongoDB's error codes reference and the server's error code list.

A collection stream receives invalidate after a drop, rename, or dropDatabase that affects the collection. A database stream receives it after dropDatabase. The event closes the cursor.

// After an invalidate event closed the stream
const next = db.orders.watch( PIPELINE, { startAfter: invalidateEvent._id } )

Interpretation: the new stream reports later changes, for example to a recreated collection with the same name. Decide whether the consumer should follow it or alert instead.

MongoDB 9.0 adds one routine interruption. A stream opened with a readPreference of primary or secondary now returns InterruptedDueToReplStateChange from getMore after an election that changes the serving node's role. Before 9.0, the cursor kept reading from the same node. The 9.0 compatibility notes say primaryPreferred, secondaryPreferred, and nearest are unaffected.

Frequently asked questions

Key takeaways

Run MongoDB change streams in production with JusDB

JusDB's MongoDB consulting team can review your change stream consumers, check oplog windows and pre-image retention against your recovery needs, and write the restart and re-copy runbook for your replica sets or sharded clusters.

Bring your topology and versions, the consumers and connectors that read change streams, and how long each one may be down. Your team approves every production change.

Related reading

• MongoDB replica set operations: adding members, failover, and lag diagnosis
• MongoDB oplog sizing and the replication window
• MongoDB 8.0 replication: new features, improvements, and what changed
• MongoDB documentation: Change Streams