StarRocks Shared-Data vs Shared-Nothing: Choosing a Deployment Mode

StarRocks runs in one of two deployment modes, and you pick it once. The FE setting run_mode is either shared_nothing (the default) or shared_data, and the StarRocks 4.1 parameter reference is blunt about changing it later: "DO NOT change run_mode after the cluster is deployed. Otherwise, the cluster fails to restart." Shared-nothing keeps table data on backend (BE) disks. Shared-data keeps it in object storage and runs stateless compute nodes (CNs) with a local cache. The choice decides cold-query latency, compaction behavior, failure cost, backups, and how fast you can add compute. This guide compares both modes for StarRocks 3.5 to 4.1 and ends with a pilot runbook.

Scope and review date: November 2026. This post covers open-source StarRocks 3.5, 4.0, and 4.1; the newest patches are 4.1.7 (26 October 2026) and 4.0.17 (24 October 2026). Commands and defaults were checked against the StarRocks documentation on 9 November 2026, not run in a lab; rehearse them on a non-production cluster.

What is the difference between shared-nothing and shared-data StarRocks?

Both modes use the same frontend (FE) nodes for metadata, planning, and scheduling. The difference is the backend. The StarRocks architecture page describes CNs as "stateless compute nodes that perform all the functions of BEs, except for the storage of data."

Shared-data arrived in StarRocks 3.0.0 (28 April 2023). The shared-data feature support page lists three hard limits: no mixed deployment, no conversion between modes in either direction, and no heterogeneous CNs ("the hardware specifications of all CN nodes within a cluster must be the same").

-- Works on any version: shows the mode the FE was started with
ADMIN SHOW FRONTEND CONFIG LIKE 'run_mode';

-- StarRocks 4.1 adds a global read-only variable for the same answer
SELECT @@run_mode;

Interpretation: the value is shared_nothing or shared_data. Check it before applying tuning advice. The @@run_mode variable is listed in the 4.1.0 release notes; on 3.5 and 4.0, use the ADMIN SHOW form.

How does shared-data store table data in object storage?

A shared-data cluster writes table data through a storage volume: a named object, created with CREATE STORAGE VOLUME, that holds the location and credentials of a bucket or HDFS path. Supported types are S3 (AWS S3 and S3-compatible stores such as GCS in S3 mode, MinIO, and Ceph), AZBLOB, ADLS2, GS, and HDFS. Its tables are "cloud-native tables" (type CLOUD_NATIVE).

The cluster needs a default storage volume before you can create databases, because StarRocks also stores system statistics there. From v3.4.1, the FE setting enable_load_volume_from_conf defaults to false, so new clusters create the volume in SQL rather than from fe.conf.

# fe/conf/fe.conf (set before the first start; immutable afterwards)
run_mode = shared_data
# cloud_native_meta_port = 6090   (default; change only if the port is taken)

# be/conf/cn.conf on every CN: the Data Cache directory
# Multiple disks are separated by semicolons; empty disables the cache
storage_root_path = /nvme1/starrocks;/nvme2/starrocks

Interpretation: run_mode is read once, at the first start. On CNs, storage_root_path is the cache location, not the data location, and the cached files sit under a starlet_cache subdirectory.

CREATE STORAGE VOLUME def_volume
TYPE = S3
LOCATIONS = ("s3://analytics-sr-data")
PROPERTIES
(
    "enabled" = "true",
    "aws.s3.region" = "us-west-2",
    "aws.s3.endpoint" = "https://s3.us-west-2.amazonaws.com",
    "aws.s3.use_aws_sdk_default_behavior" = "false",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.enable_partitioned_prefix" = "true"
);
SET def_volume AS DEFAULT STORAGE VOLUME;

Interpretation: this follows the S3 example in the shared-data deployment guide, switched to an instance profile instead of access keys. Partitioned Prefix (v3.2.4+, S3 type only) spreads objects across aws.s3.num_partitioned_prefix sub-paths (default 256) to raise request throughput. It needs a bucket-root location and cannot be changed after creation.

How does the local data cache change query latency?

In shared-nothing, a BE scans its own disks. In shared-data, a CN first looks in the Data Cache (memory, then local disk) and reads from object storage only on a miss. The architecture page states: "When the cache is hit, query performance is comparable to that of the shared-nothing architecture."

Data Cache replaced the older File Cache from v3.1.7 and v3.2.3 and loads remote data in blocks "on the order of MBs" instead of whole files. Two table properties control it: datacache.enable (default true) writes loaded data to the cache as well as object storage, and datacache.partition_duration sets how far back a partition counts as hot.

-- Collect a query profile for this session, then rerun the slow query
SET enable_profile = true;

Interpretation: the shared-data FAQ lists four usual causes of a slow shared-data query: cache misses, insufficient compaction, too few tablets for parallelism, and a datacache.partition_duration that is too short. A high PhySegmentsCount points at compaction instead.

How do you size the data cache on compute nodes?

There is no published formula, so size from your working set: the compressed size of the partitions your frequent queries touch, divided by the number of CNs, plus headroom for loads and compaction output, which also land in the cache.

From v3.4.0 a unified Data Cache instance serves both cloud-native tables and external catalogs (datacache_unified_instance_enable, default true). Its limits are datacache_disk_size (per disk, as a percentage such as 80% or a size such as 2T) and datacache_mem_size. If you set no disk size, automatic scaling evicts above 90% disk usage and grows the cache while usage stays under 60%.

-- Current cache settings on every CN
SELECT BE_ID, NAME, VALUE FROM information_schema.be_configs
WHERE NAME IN ('storage_root_path', 'datacache_disk_size',
               'datacache_mem_size', 'datacache_enable');

-- Raise the disk quota on all CNs now; also edit cn.conf, or it reverts on restart
UPDATE information_schema.be_configs SET VALUE = '2T'
WHERE NAME = 'datacache_disk_size';

Interpretation: the Data Cache documentation warns that dynamic changes are not persisted. Keep the WHERE clause, or you change every row.

-- Keep 30 days of partitions eligible for re-caching after eviction
CREATE TABLE sales.orders (
    order_date DATE NOT NULL,
    order_id   BIGINT NOT NULL,
    region     VARCHAR(32),
    revenue    DECIMAL(18, 2)
)
DUPLICATE KEY(order_date, order_id)
PARTITION BY date_trunc('day', order_date)
DISTRIBUTED BY HASH(order_id)
PROPERTIES (
    "datacache.enable" = "true",
    "datacache.partition_duration" = "30 DAY"
);

-- Warm a known hot range before a dashboard rush (v3.3+)
CACHE SELECT order_date, region, revenue FROM sales.orders
WHERE order_date >= '2026-10-01';

Interpretation: with no storage_volume property, the table uses the default volume. Data older than partition_duration is still queryable, but once evicted it is not re-cached. CACHE SELECT is synchronous, warms one table per statement, and returns TOTAL_CACHE_USAGE, which tells you whether the cache is large enough. Use NVMe disks for the cache.

How do ingestion and compaction differ between the modes?

Every load creates a new data version, and compaction merges versions into larger files. In shared-nothing, each BE writes and compacts the replicas on its own disks. In shared-data, a load writes segment files to object storage (and to the cache when datacache.enable is on), commits, and publishes a new version. The shared-data compaction guide describes an "FE-controlled compaction mechanism": the leader FE computes a compaction score per partition, picks the highest, and sends tablet-level subtasks to CNs.

That score is also an ingestion brake. Below lake_compaction_score_selector_min_score (default 10), compaction is considered done. Above lake_ingest_slowdown_threshold (default 100), StarRocks slows commits for that partition. Above lake_compaction_score_upper_bound (default 2000), it rejects loads to that partition. Frequent small loads and too many buckets are the documented causes of small-file build-up, so batch loads at the source (Routine Load intervals, Flink checkpoints) before adding CNs.

-- Partitions furthest behind on compaction
SELECT DB_NAME, TABLE_NAME, PARTITION_NAME, AVG_CS, MAX_CS
FROM information_schema.partitions_meta
ORDER BY MAX_CS DESC LIMIT 10;

-- Recent compaction transactions, with errors and a profile (v3.2.12, v3.3.4+)
SHOW PROC '/compactions';

Interpretation: the compaction guide treats MaxCS below 10 as healthy, above 100 as high, and above 500 as needing intervention. A large read_remote_mb in the profile means compaction is missing the cache.

Compaction and queries share the CNs, so size them for both. Per-CN threads are compact_threads (default 4; the guide suggests 25% of CPU cores), and cluster-wide concurrency is lake_compaction_max_tasks (default -1, meaning 16 tasks per live CN). From v4.0, File Bundling (file_bundling, on by default for new tables only) bundles files to cut object storage API calls.

How do scaling, node failure, and disaster recovery differ?

Adding a BE triggers tablet rebalancing, which copies data. Adding a CN copies nothing: CNs "can be added or removed on demand within seconds", per the architecture page. The cost moves to the cache: the shared-data FAQ lists "tablet migration due to elastic scaling" as a cause of cache misses. From v3.5.1, Cache Sharing lets a CN fetch cached blocks from peers before going to object storage.

-- Add or remove compute (CNs cannot be decommissioned, only dropped)
ALTER SYSTEM ADD COMPUTE NODE "10.0.1.24:9050", "10.0.1.25:9050";
ALTER SYSTEM DROP COMPUTE NODE "10.0.1.25:9050";

-- Cache Sharing during scaling (v3.5.1+)
ADMIN SET FRONTEND CONFIG ("enable_trace_historical_node" = "true");
SET GLOBAL enable_datacache_sharing = true;

Interpretation: 9050 is the default heartbeat_service_port. The ALTER SYSTEM reference notes that ALTER SYSTEM DECOMMISSION does not apply to CNs. enable_datacache_sharing defaults to true, so the FE setting (default false) is the real switch.

Failure handling follows from where data lives. In shared-nothing, durability is replication_num (default 3): when a BE is dropped, StarRocks re-creates its tablets from surviving replicas. In shared-data, a lost CN loses only cache, because the data is in the object store; StarRocks does not keep multiple replicas of cached data. FE high availability is identical in both modes: three or more Follower FEs.

-- Shared-data DR (beta, v3.4.2+): keep a cluster snapshot in object storage
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME def_volume;
SELECT * FROM information_schema.cluster_snapshots;

Interpretation: the Cluster Snapshot guide explains why this exists: CN data is in object storage, but FE metadata stays local. The snapshot puts both in the bucket every 600 seconds by default and keeps only the latest. The backup guide states that "Shared-data StarRocks clusters do not support data BACKUP and RESTORE".

Can you convert a shared-nothing cluster to shared-data?

Not in place. The run_mode reference says "The transformation from a shared-nothing cluster to a shared-data cluster or vice versa is not supported", and changing the setting stops the FE from restarting. Moving modes means building a second cluster and copying data into it. The Cross-cluster Data Migration Tool supports these paths as of November 2026:

The last row makes the move one-way.

-- On the TARGET cluster, before starting the tool (and in fe.conf, to survive restarts)
ADMIN SET FRONTEND CONFIG ("enable_legacy_compatibility_for_replication" = "true");

wget https://releases.starrocks.io/starrocks/starrocks-cluster-sync.tar.gz
tar -xvzf starrocks-cluster-sync.tar.gz && cd starrocks-cluster-sync

# Shared-nothing sources need the cluster token, read on a source FE
cat fe/meta/image/VERSION | grep token

vi conf/sync.properties   # source/target FE hosts, token, include_data_list
./bin/start.sh

Interpretation: the tool must reach every FE, and the target cluster every source BE. It keeps syncing while the source takes writes and never stops on its own. Watch Sync job progress in log/sync.INFO.log. When it holds at 100%, stop writes to the source, confirm it stays at 100%, stop the tool, repoint applications, and turn legacy compatibility off. It copies tables, logical views, and materialized view definitions (not their data). Do not load into the target meanwhile.

How do you pilot a shared-data cluster before committing?

Because the mode is fixed, pilot on a parallel cluster with a copy of real data and the same queries.

Which StarRocks deployment mode should you choose?

Choose from your latency target, how much compute demand varies, and how much data is cold. For cost, model your own bill: object storage request charges matter as much as storage.

Shared-nothing fits when:

• Latency on cold data matters more than elasticity. StarRocks calls it "well-suited for scenarios that pursue optimal query performance."
• You run on-premises or on fixed hardware without a reliable S3-compatible store.
• You need features the shared-data column still marks "To be supported", such as vector indexes, row store (hybrid row-column storage), or BACKUP and RESTORE.
• Compute demand is steady, so a cache adds little.

Shared-data fits when:

• Data grows faster than query load and most history is rarely read, so one copy in object storage can cost less than three replicas on block storage.
• Query load is spiky (month-end reporting, batch BI windows) and adding CNs in seconds is worth a cold cache.
• You run on Kubernetes and want HPA-driven CN scaling.
• Your hot set fits a sensible NVMe cache per CN, so most queries hit the cache.

For new shared-data clusters, the feature support page recommends the latest patch of v3.2 or later; in practice 4.0 or 4.1.

Frequently asked questions

Key takeaways

Choose and run your StarRocks deployment mode with JusDB

JusDB's StarRocks consulting team can review your workload against both modes, help plan and measure a shared-data pilot, and run a shared-nothing to shared-data migration with the cross-cluster tool, including cache sizing and compaction checks.

Bring your version, topology, load rates, slowest queries, and latency targets. Your team approves every production change.

Related reading

• StarRocks vs ClickHouse: architecture, table models, and when to choose each
• StarRocks 4.1: adaptive tablets, range distribution, and the upgrade trade-offs
• StarRocks monitoring and alerting: the production guide
• StarRocks documentation: Feature Support for Shared-data Clusters