Sail
activeby LakeSail · since 2024 · also known as LakeSail, pysail
Rust-native, Spark Connect-compatible engine for SQL, DataFrame and streaming jobs without the JVM.
Overview
Sail, from LakeSail, is a distributed compute engine written in Rust on Apache DataFusion that speaks the Spark Connect protocol, so existing PySpark and Spark SQL code can run against it without rewrites. It starts instantly, runs locally from a pip install or on Kubernetes, and reads and writes Delta Lake and Iceberg on object storage. LakeSail publishes TPC-H and ClickBench results showing large speed and cost gains over JVM Spark. It is an attractive option for cutting Spark infrastructure cost; Spark API coverage is still growing, so check your workloads.
Strengths & trade-offs
Strengths
- +Drop-in Spark Connect compatibility for PySpark and Spark SQL code
- +Rust-native: no JVM tuning, instant startup, low memory overhead
- +Large published speedups and cost savings versus JVM Spark
- +Reads and writes Delta Lake and Iceberg
- +Same engine on a laptop and on Kubernetes
Trade-offs
- −Spark API and connector coverage is not yet complete
- −Young project with limited production references
- −Benchmarks are vendor-published
Great for
- +Cutting cost of PySpark ETL
- +Local Spark development without a JVM
- +Lakehouse batch jobs on Delta/Iceberg
Think twice for
- −Jobs depending on Spark internals or RDD APIs
- −OLTP
- −Teams needing a managed service
Capabilities
Native or official first-party support · 5 of 25 tracked
Scale & consistency
- Comfortable data size
- Large · ≤ 1 PB
- Horizontal scaling
- Native
- Multi-region
- No
- Transactions
- Limited ACID
- Consistency
- Strong
- Schema
- Flexible
- Architecture
- Distributed, Single node, Client server
- Query languages
- Spark SQL, Python (PySpark DataFrame API)
Runs locally or as a distributed cluster on Kubernetes.
Deployment
Ecosystem
Alternatives to Sail
Spark's SQL and DataFrame engine for large-scale batch, streaming and lakehouse processing.
sail vs apache-spark-sql →Extensible Rust query engine library on Apache Arrow for building databases and data systems.
sail vs apache-datafusion →Serverless SQL warehouses on the Databricks lakehouse, powered by the Photon C++ engine.
sail vs databricks-sql →In-process columnar SQL database: the SQLite of analytics, running inside your app or notebook.
sail vs duckdb →Fast distributed SQL query engine for federated analytics across lakes and databases.
sail vs trino →Distributed SQL query engine from Meta for interactive queries over data lakes and many sources.
sail vs presto →Sail FAQ
What is Sail?
Sail is a relational database by LakeSail, first released in 2024. Sail, from LakeSail, is a distributed compute engine written in Rust on Apache DataFusion that speaks the Spark Connect protocol, so existing PySpark and Spark SQL code can run against it without rewrites. It starts instantly, runs locally from a pip install or on Kubernetes, and reads and writes Delta Lake and Iceberg on object storage. LakeSail publishes TPC-H and ClickBench results showing large speed and cost gains over JVM Spark. It is an attractive option for cutting Spark infrastructure cost; Spark API coverage is still growing, so check your workloads.
Is Sail open source?
Yes. Sail is open source. License: Apache License 2.0.
Does Sail support ACID transactions?
Partly. Sail offers limited ACID guarantees (for example, single-record or single-partition transactions), with strong consistency.
Does Sail scale horizontally?
Sail scales horizontally out of the box, with built-in sharding or partitioning. It is comfortable at large scale (≤ 1 PB).
How can I run Sail?
You can self-host it.
What is Sail best for?
Sail is a strong fit for cutting cost of PySpark ETL, local Spark development without a JVM and lakehouse batch jobs on Delta/Iceberg. Think twice for jobs depending on Spark internals or RDD APIs.
What are the best alternatives to Sail?
Common alternatives to Sail include Apache Spark SQL, Apache DataFusion, Databricks SQL, DuckDB and Trino. The right choice depends on your workload, scale and hosting; WhichDB's smart match ranks them against your requirements.
How popular is Sail?
Sail ranks #231 of 844 databases on the WhichDB Score (14.5/100), and #74 among relational databases. The score combines public adoption signals such as package downloads, GitHub activity, job posts and developer discussion.