Apache Paimon
activeby Apache Software Foundation · since 2023 · also known as Paimon, Flink Table Store
Streaming lakehouse table format with LSM-based upserts and changelogs, born in Apache Flink.
Overview
Apache Paimon (formerly Flink Table Store) is a lake table format designed for real-time updates: each bucket is an LSM tree of Parquet or ORC files, so primary-key tables absorb high-rate upserts and deletes from Flink CDC while producing a changelog that downstream streaming jobs can consume. It offers snapshots, time travel, tags and branches, and is readable by Flink, Spark, Trino, StarRocks, Doris and Python. Paimon fits streaming lakehouse pipelines, especially CDC into the lake; Iceberg has broader engine support for mostly-append batch data.
Strengths & trade-offs
Strengths
- +LSM-based primary-key tables handle high-rate upserts from CDC
- +Native changelog output enables incremental streaming pipelines
- +Snapshots, tags, branches and time travel
- +Readable by Flink, Spark, Trino, StarRocks and Doris
- +Can expose tables in Iceberg-compatible metadata
Trade-offs
- −A table format, not a query engine: needs Flink/Spark/Trino on top
- −Compaction tuning matters for read performance
- −Smaller engine ecosystem than Iceberg or Delta
Great for
- +Streaming CDC from databases into the lake
- +Real-time lakehouse with Flink
- +Incremental ETL with changelogs
- +Partial-update wide tables
Think twice for
- −Standalone querying without an engine
- −OLTP
- −Batch-only shops standardized on Iceberg
Capabilities
Native or official first-party support · 8 of 25 tracked
Scale & consistency
- Comfortable data size
- Massive · Multi-PB
- Horizontal scaling
- Native
- Multi-region
- No
- Transactions
- Full ACID
- Consistency
- Strong
- Schema
- Strict
- Architecture
- Distributed, Shared storage
- Query languages
- SQL (via Flink, Spark, Trino, StarRocks, Doris)
Scales with the compute engine and object storage; bucketed tables parallelize writes.
Deployment
Ecosystem
Alternatives to Apache Paimon
Columnar storage engine for fast analytics on fast-changing data, paired with Impala or Spark.
apache-paimon vs apache-kudu →MPP OLAP database with a fast vectorized engine, real-time upserts and native lakehouse queries.
apache-paimon vs starrocks →Serverless SQL warehouses on the Databricks lakehouse, powered by the Photon C++ engine.
apache-paimon vs databricks-sql →Spark's SQL and DataFrame engine for large-scale batch, streaming and lakehouse processing.
apache-paimon vs apache-spark-sql →Fast distributed SQL query engine for federated analytics across lakes and databases.
apache-paimon vs trino →A Rust stream processor with SQL pipelines, now owned by Cloudflare and powering Cloudflare Pipelines.
apache-paimon vs arroyo →Apache Paimon FAQ
What is Apache Paimon?
Apache Paimon is a columnar / OLAP database by Apache Software Foundation, first released in 2023. Apache Paimon (formerly Flink Table Store) is a lake table format designed for real-time updates: each bucket is an LSM tree of Parquet or ORC files, so primary-key tables absorb high-rate upserts and deletes from Flink CDC while producing a changelog that downstream streaming jobs can consume. It offers snapshots, time travel, tags and branches, and is readable by Flink, Spark, Trino, StarRocks, Doris and Python. Paimon fits streaming lakehouse pipelines, especially CDC into the lake; Iceberg has broader engine support for mostly-append batch data.
Is Apache Paimon open source?
Yes. Apache Paimon is open source. License: Apache License 2.0.
Does Apache Paimon support ACID transactions?
Yes. Apache Paimon supports full ACID transactions, with strong consistency.
Does Apache Paimon scale horizontally?
Apache Paimon scales horizontally out of the box, with built-in sharding or partitioning. It is comfortable at massive scale (Multi-PB).
How can I run Apache Paimon?
You can self-host it and embed it in your application.
What is Apache Paimon best for?
Apache Paimon is a strong fit for streaming CDC from databases into the lake, real-time lakehouse with Flink and incremental ETL with changelogs. Think twice for standalone querying without an engine.
What are the best alternatives to Apache Paimon?
Common alternatives to Apache Paimon include Apache Kudu, StarRocks, Databricks SQL, Apache Spark SQL and Trino. The right choice depends on your workload, scale and hosting; WhichDB's smart match ranks them against your requirements.
How popular is Apache Paimon?
Apache Paimon ranks #237 of 844 databases on the WhichDB Score (14.2/100), and #25 among columnar / OLAP databases. The score combines public adoption signals such as package downloads, GitHub activity, job posts and developer discussion.