How Notion Handles 200 Billion Data Objects
Transition from PosgreSQL-only solutions to your own DataLake to separate the read load for the needs of analytics and AI.
Introduction
If you used Notionyou know that it allows you to do almost everything – take notes, plan, make reading lists and manage projects.
Notion is a flexible tool. You can customize the space until you feel you have achieved what you want.
Everything is in Notion is a block – text, images, lists, database rows and even pages.
These dynamic units can be converted into blocks of other types or moved freely within Notion.
Blocks are LEGO from Notion.
Postgres ruled them all.
Notion initially stored all blocks in a Postgres database. In 2021, there were more than 20 billion of them. Now, the number of blocks has grown to 200 billion.
Until 2021, everything fit into a single Postgres instance.
The database is currently split into 480 logical segments, which are distributed across 96 Postgres instances, so that each is responsible for 5 segments.
Postgres handled everything from online user traffic to offline analytics and machine learning.
Realizing the great demands for analytics and AI, it was decided to create our own infrastructure for them.
Fivetrans and Snowflake
First, in 2021, a simple ETL was created that used Fivetran to transfer data from Postgres to Snowflake. 480 connectors were used to write 480 segments to raw Snowflake tables every hour.
Notion then combined these tables into one large one for analysis and machine learning.
But this approach had some problems when the Postgres data volume grew:
Managing 480 Fivetran connectors was a nightmare:
Notion users update blocks more often than they add new ones. This pattern of intensive updating slows down and increases the cost of processing Snowflake data.
Data consumption is becoming increasingly complex and heavy (AI workloads)
Notion has started building its own data warehouse.
Data Lake
The following requirements were set for the system:
Scalable data warehouse for storing both raw and processed data.
Fast and cost-effective data ingestion and computation for any workload. Especially with frequently updated data.
In 2022, DataLake's own architecture was introduced, the essence of which is to transfer data from Postgres to Kafka using Debezium, with subsequent transfer to Apache Hudi and S3.
Such an object store acts as an endpoint for consumers:
Spark is used to process the billions of blocks received and their updates.
Thanks to the creation of such a subsystem, it was possible to significantly reduce current costs.
Below is the main data upload pipeline:
1 Debeizum CDC connector per 1 Postgres instance.
Connectors are deployed and managed in Kubernetes on AWS (EKS)
The connector can process Postgres row changes at tens of MB/s.
One Kafka topic per 1 Postgres table.
All connectors consume data from 480 segments and write to the appropriate topic for the given table.
Apache Hudi Deltastreamer used to receive messages from Kafka and write data to S3.
Most of the data processing jobs were written in PySpark.
For more complex tasks, Scala Spark is used. Notion also uses multithreading and parallel processing to speed up the processing of 480 segments.
Win
Offloading data from Snowflake to S3 saved Notion over $1 million in 2022, with even more savings expected in 2023 and 2024.
The total time to receive and process data from Postgres has been significantly reduced: from more than a day to a few minutes for small tables and a couple of hours for large ones.
The new data infrastructure opens up additional opportunities for using analytics. It helped successfully implement new AI functionality in 2023 and 2024.
Instead of finishing
More distributed fault-tolerant systems, patterns, architectural cat, System Design interview in telegram community system_design_world.
