Current-version production checklist
This guide originated in the 2025 DuckDB ecosystem. DuckDB follows semantic versioning, and the official release calendar lists 1.5.5 as the current patch release on August 1, 2026. Pin the client and extension versions used by a production job, record SELECT version() with benchmark results, and retest plans and file compatibility before an upgrade. Throughput claims from a different machine, file layout, cache state, or DuckDB version are not an SLA.
Use the actual concurrency model
DuckDB is not limited to one writer thread. Within one process it supports concurrent writer threads using MVCC and optimistic concurrency control; appends do not conflict, while concurrent updates to the same row can return a conflict error. What DuckDB does not automatically support is multiple processes writing to the same database file. Multiple processes can open the file read-only, or the application must serialize cross-process writes itself.
Give each worker thread its own connection instead of sharing the Python module's global connection. Bound retries for optimistic conflicts and retry the complete transaction, not only its final statement. If independent services need high-concurrency writes to one shared database, place a service or queue in front of a single owning process, partition ownership, or choose a client-server system. The official DuckDB concurrency guide should be the source of truth for the version you deploy.
Put hard limits around memory and spill
DuckDB can spill larger-than-memory sorts, joins, and aggregations to a temporary directory, but the default is not a capacity plan. Set memory_limit, threads, temp_directory, and max_temp_directory_size for the host or container. Leave headroom because some vectors, result objects, and complex aggregate state are allocated outside the buffer manager and can exceed memory_limit. Put the temporary directory on monitored local storage with enough IOPS and space; an exhausted spill volume or an operator that cannot spill cancels the query.
SET threads = 4;
SET memory_limit = '8GB';
SET temp_directory = '/var/tmp/duckdb-spill';
SET max_temp_directory_size = '100GB';
EXPLAIN ANALYZE
SELECT event_type, count(*)
FROM read_parquet('s3://analytics/events/date=2026-08-01/*.parquet')
WHERE event_ts >= TIMESTAMP '2026-08-01'
GROUP BY event_type;EXPLAIN ANALYZE executes the query and reports runtime operator metrics, so use it against a safe copy or bounded partition when side effects or cost matter. For scheduled jobs, alert on process RSS, spill-directory bytes, query duration, object-store errors, and result cardinality. A job that returns quickly after reading zero files is not a successful pipeline.
Validate Parquet pruning and remote access
DuckDB automatically pushes required columns and eligible filters into Parquet scans and can skip row groups using their statistics. Performance therefore depends on file count, row-group layout, sort order, compression, selectivity, and network range requests. Inspect the plan to confirm filters reached the scan. When writing a dataset, choose row groups with enough parallel units for the intended thread count and consider sorting on frequently selective columns so min/max statistics can prune effectively. The Parquet reader documentation and Parquet layout guidance describe these tradeoffs without promising a universal file size.
For S3, use DuckDB's Secrets Manager and scoped, short-lived credentials rather than embedding keys in SQL or source code. Validate bucket region, endpoint, certificate verification, range-read permissions, listing permissions needed by globs, credential refresh, retry behavior, and egress cost. A successful query against one object does not prove that a partition glob is complete; compare discovered filenames or partition counts with the producer manifest.
Protect the transactional source
The current PostgreSQL extension documentation recommends ATTACH ... (TYPE postgres, READ_ONLY) when modifications are not intended; the older postgres_attach function is deprecated. Use a least-privilege database role, statement and connection timeouts, and a read replica when an analytical scan could contend with OLTP. PostgreSQL tables are read at query time, so repeated scans can repeatedly load the source. Materialize a bounded snapshot in DuckDB or Parquet when reproducibility and source isolation matter, and record the extraction boundary used for validation.
Go-live tests
- Restart the process and prove persistent data, extensions, secrets, and spill paths return correctly.
- Run the largest credible partition with cold caches and an enforced memory limit.
- Kill a job during a write, reopen the database, and verify the intended transaction boundary and output manifest.
- Attempt concurrent same-row updates and a second writer process so the application handles both documented failure modes.
- Reconcile row counts, null rates, min/max timestamps, and business totals against the source before publishing output.