Question 1

What Apache Pinot migrations do you handle?

Accepted Answer

Druid → Pinot (user-facing-latency requirements that Druid can't hit), ClickHouse → Pinot (multi-tenant isolation needs), realtime-to-hybrid table conversion (adding offline batch tier to existing realtime workload), self-managed → StarTree Cloud (operational outsourcing), upsert-workload migrations leveraging Pinot's primary-key support. For the engine-decision phase, see Pinot consulting.

Question 2

Druid → Pinot — when does this happen?

Accepted Answer

When the workload outgrows Druid's roll-up-segment model. Triggers: (a) user-facing analytics need sub-100ms p99 latency that Druid can't consistently deliver, (b) multi-tenant SaaS requires Pinot's tag-based server segmentation, (c) raw-row drill-down is needed (Druid roll-up is destructive). Re-modelling required: Druid time-partitioned segments → Pinot star-tree-indexed tables, supervisor tasks → LLRT consumers.

Question 3

ClickHouse → Pinot — what's the trigger?

Accepted Answer

Multi-tenant SaaS isolation. ClickHouse handles single-tenant analytics excellently but multi-tenant noisy-neighbour prevention is a project; Pinot's tag-based server segmentation does it natively. Also: user-facing dashboard p99 budgets that ClickHouse cold-cache cost can't hit. Migration: schema re-modelling (MergeTree → star-tree tables), Kafka stream re-targeting, multi-tenant table-per-tenant or shared-table-with-tenant-column design.

Question 4

Realtime-to-hybrid table conversion — what's involved?

Accepted Answer

Hybrid tables combine LLRT realtime ingestion with batch-built offline segments. Common pattern: realtime for last 7-30 days, offline for historical depth. Conversion sequence: (1) configure offline table with matching schema, (2) build Spark/Hadoop job for batch segment generation, (3) upload to deep storage, (4) configure hybrid table to federate queries across both, (5) shrink realtime retention window. Queries transparently span both segments.

Question 5

Self-managed → StarTree Cloud — economics?

Accepted Answer

StarTree Cloud handles Controller / Broker / Server / Minion operations, Helix coordination, deep storage, multi-tenancy. Wins for: fast time-to-value, multi-tenant SaaS isolation (StarTree built specifically for this), small ops teams. Self-managed wins for: sustained workloads where reserved-instance EC2 beats StarTree pricing, regulatory placement requirements, or teams with deep K8s + Helix expertise.

Question 6

Helix coordination — how to manage during migration?

Accepted Answer

Apache Helix manages segment assignment in Pinot. During migration, Helix state needs careful handling: rebalances during cutover create extra load, segment assignments need consistent ZooKeeper state, and recovery from Helix-state mismatch is a real operational risk. We design migration to avoid Helix rebalances at peak load and validate state consistency before cutover gates.

Question 7

Cutover playbook for zero-downtime?

Accepted Answer

Standard: (1) provision Pinot target with sized topology, (2) dual-load realtime stream to both source and Pinot, (3) bulk-load historical via batch segment generation, (4) wait for replica lag to settle, (5) flip reads — observe p99 latency + error rate, (6) quiesce writes on source briefly, (7) promote Pinot, (8) 24-48 hour soak, (9) decommission source. Helix state validation at each gate.

Apache Pinot Migration Services

Apache Pinot migrations we handle

Pinot migration — common questions

Ready to plan the Pinot migration?