Apache DataFusion
activeby Apache Software Foundation · since 2019 · also known as DataFusion, Arrow DataFusion
Extensible Rust query engine library on Apache Arrow for building databases and data systems.
Overview
Apache DataFusion is an embeddable SQL and DataFrame query engine written in Rust on Apache Arrow's columnar memory format. Rather than a finished database, it is a toolkit: a parser, cost-aware optimizer and multi-threaded vectorized executor with extension points for custom tables, functions, optimizer rules and execution plans. Systems such as InfluxDB 3 are built on it, and the Comet subproject accelerates Spark. DataFusion is the right choice when you are building a data product; end users looking for a ready-made database are better served by DuckDB or ClickHouse.
Strengths & trade-offs
Strengths
- +Highly extensible: custom sources, UDFs, optimizer rules and plans
- +Fast vectorized execution on Arrow with good ClickBench results
- +Safe, embeddable Rust with no runtime dependencies
- +Reads Parquet, CSV, JSON, Avro and integrates with Iceberg/Delta crates
- +Large ecosystem of systems built on it (InfluxDB 3, Comet, many startups)
Trade-offs
- −A library, not a database: no storage, server or users out of the box
- −Requires Rust expertise to build on
- −Frequent major versions with breaking API changes
Great for
- +Building custom databases and query engines
- +Embedded SQL in Rust services
- +Accelerating Spark via Comet
- +Query layer for time-series or log stores
Think twice for
- −End users wanting a ready-made database
- −OLTP
- −Non-Rust teams without engineering capacity
Capabilities
Native or official first-party support · 4 of 25 tracked
Scale & consistency
- Comfortable data size
- Medium · ≤ 10 TB
- Horizontal scaling
- Addon
- Multi-region
- No
- Transactions
- None
- Consistency
- Strong
- Schema
- Strict
- Architecture
- Embedded, In memory
- Query languages
- SQL, DataFrame API (Rust/Python), Substrait
Single-process multi-threaded; distributed execution via the Ballista subproject or the host system.
Deployment
Ecosystem
Alternatives to Apache DataFusion
In-process columnar SQL database: the SQLite of analytics, running inside your app or notebook.
apache-datafusion vs duckdb →Spark's SQL and DataFrame engine for large-scale batch, streaming and lakehouse processing.
apache-datafusion vs apache-spark-sql →Embedded, in-process OLAP engine powered by ClickHouse, used mainly from Python.
apache-datafusion vs chdb →Open-source columnar OLAP database known for very fast aggregations over billions of rows.
apache-datafusion vs clickhouse →Fast distributed SQL query engine for federated analytics across lakes and databases.
apache-datafusion vs trino →Distributed SQL query engine from Meta for interactive queries over data lakes and many sources.
apache-datafusion vs presto →Apache DataFusion FAQ
What is Apache DataFusion?
Apache DataFusion is a columnar / OLAP database by Apache Software Foundation, first released in 2019. Apache DataFusion is an embeddable SQL and DataFrame query engine written in Rust on Apache Arrow's columnar memory format. Rather than a finished database, it is a toolkit: a parser, cost-aware optimizer and multi-threaded vectorized executor with extension points for custom tables, functions, optimizer rules and execution plans. Systems such as InfluxDB 3 are built on it, and the Comet subproject accelerates Spark. DataFusion is the right choice when you are building a data product; end users looking for a ready-made database are better served by DuckDB or ClickHouse.
Is Apache DataFusion open source?
Yes. Apache DataFusion is open source. License: Apache License 2.0.
Does Apache DataFusion support ACID transactions?
No. Apache DataFusion does not provide ACID transactions; its consistency model is strong.
Does Apache DataFusion scale horizontally?
Apache DataFusion scales horizontally through an official extension, proxy or edition rather than in the core engine. It is comfortable at medium scale (≤ 10 TB).
How can I run Apache DataFusion?
You can self-host it and embed it in your application.
What is Apache DataFusion best for?
Apache DataFusion is a strong fit for building custom databases and query engines; embedded SQL in Rust services; and accelerating Spark via Comet. Think twice for end users wanting a ready-made database.
What are the best alternatives to Apache DataFusion?
Common alternatives to Apache DataFusion include DuckDB, Apache Spark SQL, chDB, ClickHouse and Trino. The right choice depends on your workload, scale and hosting; WhichDB's smart match ranks them against your requirements.
How popular is Apache DataFusion?
Apache DataFusion ranks #76 of 844 databases on the WhichDB Score (31.6/100), and #9 among columnar / OLAP databases. The score combines public adoption signals such as package downloads, GitHub activity, job posts and developer discussion.