Running valkey-search on a Valkey Cluster: The Coordinator, Index Consistency and Partial Results

You load valkey-search on a Valkey Cluster, create an index, and FT.SEARCH returns fewer documents than you stored. There is no error. Or FT.INFO idx PRIMARY fails with ERR PRIMARY option is not valid in this configuration. Both come from the same place: the search coordinator, a gRPC service inside the module that fans each query out to every shard and merges the answers. It needs its own port, a module argument that the valkey-bundle image does not pass, and timeout settings whose defaults the valkey.io pages disagree about. This guide covers valkey-search 1.2.1, the module in valkey-bundle 9.1.4 on Valkey 9.1.3: how the coordinator works, what it needs from the network, what happens on a timeout, and how to enable, monitor, and upgrade it.

Scope and review date: November 2026. This post covers valkey-search 1.2.1, the newest GA release, on self-managed Valkey Cluster 9.0 and 9.1, with notes on 1.3.0-rc2. Defaults, error strings, and port logic were checked on 10 November 2026 against the valkey.io documentation, the GitHub release notes, and the module source at the 1.2.1 and 1.3.0-rc2 tags. The commands were not run in a lab, so rehearse them on a staging cluster first.

Which valkey-search version does valkey-bundle ship?

As of November 2026, the newest GA release is valkey-search 1.2.1, published 7 July 2026. Like 1.2.0, it requires Valkey 9.0.1 or later. 1.3.0-rc2 followed on 26 October 2026 and requires Valkey 9.1.0.

In the bundle's versions.json, the 9.1.4 tag, pushed to Docker Hub with 9.1 and latest on 27 October 2026, contains Valkey 9.1.3 and search 1.2.1. The 9.0.7 and 8.1.11 images still carry search 1.0.3. Only the 9.2.0-rc4 image carries search 1.3.0-rc2, on Valkey 9.2.0-rc2.

# On every node: core version and loaded modules
valkey-cli -h 10.0.1.11 INFO server | grep valkey_version
valkey-cli -h 10.0.1.11 MODULE LIST

Interpretation: MODULE LIST names the module search, with ver packed as major × 65536 + minor × 256 + patch, so 1.2.1 reports 66049. Every node should match before you create indexes.

Why does FT.SEARCH return only one node's keys?

Because the coordinator is off. In the 1.2.1 source, use-coordinator defaults to false and is a hidden setting. Hidden settings are not registered with Valkey's CONFIG system, so the only way to turn it on is the module argument --use-coordinator, given with no value, at startup. Neither CONFIG SET nor a search.use-coordinator line in valkey.conf can set it.

Without the coordinator, a cluster node behaves like a standalone server. FT.CREATE reaches that node and its replicas, no further, and FT.SEARCH searches only that node's slots. The valkey-bundle entrypoint appends --loadmodule for every module with no arguments, and the valkey-bundle page even shows search.use-coordinator no under the comment "Enable cluster mode".

# valkey.conf on every cluster node (valkey-bundle path shown)
cluster-enabled yes
loadmodule /usr/lib/valkey/libsearch.so --use-coordinator

Interpretation: when you pass the config file as the container argument, the bundle entrypoint skips its own --loadmodule for any module whose loadmodule line it finds there, so this line wins. Do not write --use-coordinator yes. The parser treats the flag as taking no value, so it reads yes as the next argument and rejects it because it does not start with --.

After a restart, the module logs a use_coordinator: line, and the search_coordinator_* fields appear in INFO SEARCH only when the coordinator runs. If they are missing, that node searches locally only.

How does a query fan out across shards?

Per the search overview, index definitions are copied to every node, but each node indexes only the keys in its own slots. Any node that receives FT.SEARCH becomes the coordinator for that query. In the 1.2.1 source, it picks one node per shard at random, primary or replica, and sends each a gRPC request that carries a fingerprint of its index definition. It then merges the replies.

So replicas serve queries, and replication lag shows up in search results. Every query also costs one RPC per shard. For a non-vector query, each shard returns up to offset plus limit results, so LIMIT 1000 10 asks every shard for 1,010 keys. For a KNN query, each shard returns up to K. The overview warns that queries "may scale sub-linearly with increasing shard count".

The way around fan-out is a single-slot index: its name contains a hash tag, and every prefix must carry the same tag. Since 1.2.0, queries on it go to one node. A cluster-wide index has the opposite rule: no hash tag in its prefixes.

# Single-slot index: one tenant, one slot, one node per query
valkey-cli -c -h 10.0.1.11 FT.CREATE "idx:{tenant42}" ON HASH \
  PREFIX 1 "doc:{tenant42}:" SCHEMA price NUMERIC category TAG

# Cluster-wide index: every shard holds part of the data
valkey-cli -c -h 10.0.1.11 FT.CREATE products ON HASH PREFIX 1 product: \
  SCHEMA category TAG embedding VECTOR HNSW 6 \
  TYPE FLOAT32 DIM 768 DISTANCE_METRIC COSINE

Interpretation: breaking either rule fails with All PREFIX arguments must contain the same hash tag as the index or PREFIX argument(s) must not contain a hash tag. A single-slot index keeps all its data in one shard, so it suits per-tenant data, not one large catalogue.

Which ports and TLS settings does the coordinator need?

A cluster node already uses the client port and the cluster bus, which is the client port plus 10000 unless cluster-port is set (cluster tutorial). valkey-search adds a third: the Valkey port plus 20294, per the overview and GetCoordinatorPort() in src/coordinator/util.h. Port 6378 is the exception and uses an offset of 20295, so both 6378 and 6379 map to 26673. No config overrides the offset, and the module refuses to start the coordinator if the Valkey port is above 45241.

Two TLS details matter. The listener takes its base from tls-port when that is non-zero, and from port otherwise. And the gRPC channel itself is not encrypted. In both the 1.2.1 and 1.3.0-rc2 source, the server and client are built with gRPC's insecure credentials, and the server binds to [::], all addresses. We found no authentication on the service. Queries and result documents cross it in plaintext, even when clients use TLS.

# Allow gRPC only from the cluster subnet (example with ufw)
sudo ufw allow from 10.0.1.0/24 to any port 26673 proto tcp
sudo ufw deny 26673/tcp

# From each node, check every peer's coordinator port
for n in 10.0.1.11 10.0.1.12 10.0.1.13; do nc -vz -w 2 "$n" 26673; done

Interpretation: every node must reach every other node on this port, replicas included, because replicas serve queries. A failed nc means queries through that node drop a shard. On untrusted networks, encrypt below the application, for example with a VPN between hosts.

For TLS and ACLs on the client and bus ports, see hardening Valkey with ACLs, TLS and LDAP.

How consistent are index definitions across the cluster?

Eventually consistent, by design. Each primary periodically broadcasts a Merkle-tree checksum of its index definitions over the cluster bus. Nodes that see a mismatch negotiate each index with version numbers and last-writer-wins timestamps. The node that loses deletes its copy and recreates it from the winning definition, which restarts that node's backfill. Since 1.2.0, replicas no longer take part and take their metadata from their primary.

FT.CREATE and FT.DROPINDEX change the local copy first and then wait for the cluster to converge. In 1.2.1, the create check waits up to search.ft-info-timeout-ms (5000 ms by default in the source). If convergence fails, the command returns an error. The overview is explicit that "No attempt is made to undo any failed metadata mutation". The usual cause is a shard down or a network partition.

valkey-cli -c -h 10.0.1.11 FT.CREATE orders ON HASH PREFIX 1 order: \
  SCHEMA status TAG total NUMERIC
# (error) Unable to contact all cluster members

# Do not retry blindly: see whether the index exists anyway
valkey-cli -c -h 10.0.1.11 FT._LIST
valkey-cli -c -h 10.0.1.12 FT._LIST

Interpretation: this is the fan-out timeout reply in the 1.2.1 source; other failures read Inconsistent index state error found. or Communication error between nodes found. If FT._LIST shows the index, it exists and will spread once the shard is back, and a second FT.CREATE fails because the name is taken.

What happens when a shard times out or fails?

This is where the documentation contradicts itself. Four pages and the source give different defaults:

The source wins. Out of the box, FT.SEARCH, FT.AGGREGATE, and FT.INFO run as SOMESHARDS and INCONSISTENT. In the 1.2.1 fan-out code, a shard whose RPC returns an error is left out, and the reply carries no marker. The command fails only if every shard fails.

# Completeness required: fail instead of dropping a shard
valkey-cli -c -h 10.0.1.11 FT.SEARCH products "@category:{shoes}" \
  ALLSHARDS CONSISTENT TIMEOUT 2000 LIMIT 0 20

# Or change the default for every query on a node
valkey-cli -h 10.0.1.11 CONFIG SET search.enable-partial-results no

Interpretation: with ALLSHARDS, the first failed shard fails the query. The index fingerprint check always runs on each shard, and CONSISTENT adds a slot-ownership check. With CONSISTENT, a mismatch returns Index or slot consistency check failed; without it, the mismatched shard is treated like any failed shard. CONFIG SET is per node, so apply it everywhere and in your config files.

Three timeouts apply (defaults from the 1.2.1 source):

• search.default-timeout-ms: 50000, used when a query gives no TIMEOUT. TIMEOUT is capped at 60000; 1.3 adds search.max-timeout-ms for the cap.
• search.coordinator-query-timeout-secs: the gRPC deadline per shard RPC. 25 in the source; the monitoring page says 120.
• search.ft-info-timeout-ms: 5000 (2500 per RPC) for FT.INFO fan-out and the FT.CREATE check.

Reading the 1.2.1 source, partial results cover shards that return an error inside the query window. If the coordinator's own query timeout fires first, the client gets Search operation cancelled due to timeout whatever the setting. Test both paths on staging.

How do you size and monitor search on a cluster?

Size per node, not per cluster. Each primary indexes its slots, and each replica builds and holds its own copy of the index from the replication stream. Backfill runs on every node independently. Reader and writer threads default to the physical core count. There is no vendor formula for index memory, so load a representative sample and measure. search_used_memory_bytes in INFO SEARCH gives the module total per node. FT.INFO gives user_indexed_memory per attribute, summed across primaries with PRIMARY and across all nodes with CLUSTER.

The search observables page drops the search_ prefix that INFO prints; the monitoring page keeps it. Alert on the prefixed names.

valkey-cli -h 10.0.1.11 INFO SEARCH | grep -E \
  'coordinator_client_search_index_partition_failure_count|get_global_metadata_failure_count|background_indexing_status|failure_requests_count'

Interpretation: a rise in the partition failure count means this node, as coordinator, missed a shard's answer; under SOMESHARDS that query returned incomplete results. A rising metadata failure count points to metadata sync trouble. Alert on the rate of these counters; Valkey monitoring with Prometheus covers the exporter side.

How do you enable search on an existing cluster safely?

The coordinator setting is read only at startup, so enabling it is a rolling restart: replicas first, then fail over each shard and restart the old primary.

Coming from Redis Stack? See moving off Redis Stack to valkey-bundle for which indexes to rebuild.

What breaks during upgrades, node failures, and slot migrations?

Module upgrades

Upgrade the core first: each search release refuses to load on a core older than its minimum. The bundle's 9.0 to 9.1 move takes search from 1.0.3 to 1.2.1. Mixed versions are allowed during a roll, but per src/version.h, a node older than a metadata message's minimum module version drops the whole message. Do not use features newer than the oldest node's release, such as 1.3 FLOAT16 vectors, until every node is upgraded.

# Downgrade check: an older module refuses newer index RDB data
grep "ValkeySearch RDB contents require minimum version" /var/log/valkey/valkey.log

Interpretation: this error, from the 1.2.1 loader, means the RDB holds index data that needs a newer module. Keep an RDB written before each module step, because an older module can refuse one written after it. On 1.3, leave search.emulate-release at its default until every node is upgraded, because its fixes change query behaviour.

A node goes down

Until the cluster marks the node failed (cluster-node-timeout defaults to 15000 ms per valkey.conf), random target selection can still pick it. The 1.2.1 gRPC client sets waitForReady, so a call to an unreachable peer waits until its 25-second deadline, which is shorter than the 50-second query timeout. Our reading of the source: such a query stalls for about 25 seconds, then returns without that shard. Set an explicit TIMEOUT on latency-sensitive queries. After failover, the coordinator refreshes its topology every 250 ms (search.cluster-map-expiration-ms).

A slot is migrating

1.3.0-rc1 fixes results that included keys in slots the node does not own, which double counted during slot migration (#921). On 1.2.1, expect duplicate hits or inflated totals during a reshard. Only CONSISTENT checks slot ownership and rejects such queries. Inside MULTI/EXEC or Lua, 1.2.1 rejects FT.SEARCH in cluster mode with MULTI/EXEC or Lua script are not supported in CME mode. 1.3.0-rc1 allows it for a single-slot index that the local node owns (#866).

Frequently asked questions

Key takeaways

Run valkey-search on Valkey Cluster with JusDB

JusDB's Valkey Cluster team can review how valkey-search is loaded and networked on your cluster, check your timeout and partial-result settings against what your application expects, and plan a rolling enablement or module upgrade with a rollback point.

Bring your Valkey and valkey-bundle versions, your module load lines, your index definitions, and the queries that matter most. Your team approves every production change.

Related reading

• Valkey performance tuning: memory, persistence, and cluster config
• Valkey 9.1: database ACLs, Lua and I/O threading, and how to upgrade
• Embedding storage at scale: PostgreSQL, Redis, and dedicated vector databases
• Valkey documentation: Valkey Search overview