Which data pipeline architecture should you use?
Here's an overview of the data pipeline architectures you can use today.
Data is important to any application and is needed to develop efficient pipelines for delivering and managing information. Typically, a data pipeline is created when you need to process data throughout its lifecycle. A data pipeline can start where data is generated and stored in any format. A data pipeline can provide data analysis, business use, long-term storage, and machine learning model training.
Data is extracted, processed and transformed in several stages depending on the requirements of the downstream system. All processing and transformation stages are defined in the data pipeline. Depending on the requirements, pipelines can be either simple, consisting of one link, or complex, including several stages of data conversion and processing.
How to choose a conveyor design pattern?
When choosing a data pipeline design pattern, there are various design factors to consider. Here are some of them:
- Selecting data source formats.
- What stacks to use.
- Selecting data transformation tools.
- Select a data extraction option: Extract-Transform-Load (ETL), Extract-Load-Transform (ELT), or Extract-Transform-Load-Transform (ETLT).
- How to manage changed data.
- How changes are recorded.
Data sources can contain different types of information. Building a pipeline is also key dependent on the technology stack and toolkits we use. Enterprise development environments are complex to handle and require multiple and complex techniques to take changed data and merge it with target data.
I have already mentioned that most often the requirements for the pipeline and the ways in which the above processes are interconnected depend on the downstream systems. The processing stages and data flow sequence are the main factors influencing pipeline design. Each stage may include one or more data inputs, and data output may occur in one or more stages. Processing between input and output may involve simple or complex transformation steps. I strongly recommend keeping the design simple and modular to ensure a clear understanding of the steps and transformations taking place. Additionally, the pipeline's simple and modular design makes it easier for the development team to implement development and deployment cycles. This also makes it easier to debug and troubleshoot the pipeline when problems arise.
Main components of the conveyor:
- Data source
- Processing
- Target storage
Source data can be transactional applications, files collected from users, and data retrieved through an external API. Processing of source data can be as simple as one copying step or as complex as several transformations and merging with data from other sources. The system into which the prepared information arrives may require processed data resulting from transformation (for example, after changing the data type or extracting it), as well as searching and updating data from other systems. A simple data pipeline can work by copying data from a source to a target system without any modification. A complex data pipeline may involve multiple stages of transformation, searching, updating, calculating KPIs, and storing data in multiple target systems for various reasons.
Source data can be presented in various formats. Each of them requires an appropriate architecture and tools for processing and transformation. A typical data pipeline may require multiple types of data, which can be represented in any of the following formats:
- Batch Data: A tabular information file (CSV, JSON, AVRO, PARQUET and …) in which data is collected according to a certain threshold or frequency using conventional batch or micro-batch processing. Modern applications typically generate continuous data. For this reason, microbatch processing is the preferred option for collecting data from sources.
- Transactional data: Application data such as RDBMS (relational data), NoSQL, Big Data.
- Streaming Data: Real-time applications using Kafka, Google Pub/Sub, Azure Stream Analytics, or Amazon Stream Data. Streaming data applications can communicate in real time and exchange messages to meet demands. When designing an enterprise architecture, real-time data and stream processing is a very important design component.
- Flat files are PDF files or other non-tabular formats that contain data to be processed. For example, medical or legal documents from which information can be extracted.
The target data is determined depending on the requirements and needs for further processing. Typically, target data is created to support multiple systems. In the Data Lake concept, data is processed and stored in such a way that analytical systems can obtain information and AI/ML processes can use the data to build predictive models.
Architectures and examples
Various architectural designs are examined to show how source data is extracted and transformed into target data. The goal is to explain general approaches, but it is important to remember that each use case can be very varied and unique to the client and requires specific consideration.
Data pipeline architecture can be divided into logical and platform layers. Logical design describes how data is processed and transformed from source to target. Platform design focuses on the implementation and tools that each environment needs, and this depends on the vendor and tooling available in the platform. GCP, Azure or Amazon have different sets of tools for transformation, while the goal of logical design remains the same (data transformation) no matter which provider is used.
Here is the logical diagram of the data warehouse pipeline:
Here is the logic diagram of the Data Lake pipeline:
Depending on subsequent requirements, typical architectural designs can be implemented in greater detail to address multiple use cases.
Implementations on specific platforms may vary depending on the choice of tooling and the skills of the developer. Below are some examples of GCP implementations for common data pipeline architectures.
- Batch ETL pipeline in GCP – the source can be files that need to be loaded into the Business Intelligence (BI) analytics engine. Cloud storage is the data transfer medium inside GCP and then Dataflow used to load data into the target storage BigQuery. The simplicity of this approach makes the pattern reusable and effective for simple transformations. On the other hand, if you need to build a complex pipeline, then this approach is not very effective and efficient
- The data analysis pipeline is a complex chain that includes pipelines for both batch and stream data input. The processing is complex and many tools and services are used to transform the data into storage and AL/ML access point for further processing. Enterprise data analytics solutions are complex and require multi-step data processing. Design complexity can increase project time and cost, but each component must be carefully analyzed and designed to achieve business goals.
- The Machine Learning Data Pipeline in GCP is an end-to-end project that allows customers to leverage all of GCP's native services to build and process machine learning processes. For more information, see Creating a Machine Learning Pipeline
GCP platform diagrams are created in Google Cloud Developer Architecture.
How to choose an architecture for your data pipeline?
There are many approaches to designing and implementing data pipelines. The main thing is to choose the one that meets your requirements. New technologies are emerging that provide more reliable and faster implementations of data pipelines.
is a new service that offers a new approach to data entry. BigLake is a storage engine that unifies data stores, allowing BigQuery and free frameworks such as Spark to access data with fine-grained access control. BigLake delivers improved query performance on multi-cloud storage and open formats such as Apache Iceberg.
Another important factor when choosing a data pipeline architecture is cost. Creating a cost-effective solution is one of the main factors when choosing a project. Typically, stream and real-time processing pipelines are more expensive to build and operate compared to batch models. There are times when budget drives decisions about how to design and build a platform. Detailed knowledge of each component and the ability to conduct a cost analysis of the solution in advance are very important for the correct choice of architecture. GCP provides cost calculatorwhich can be used in such cases.
Do you really need real-time analytics or is a near-real-time system sufficient? This is important to consider when designing a flow conveyor. Are you building a cloud solution or migrating an existing one from an on-premises environment? All of these questions are important to designing the right data pipeline architecture.
Don't ignore data volume when designing your data pipeline. Scalability of the design and services used in the platform is another very important factor that must be considered when developing and implementing a solution. Big data is becoming larger and larger and increasing processing power. Data storage is a key element of a data pipeline architecture. In reality, there are many variables that can help in properly designing a platform. The volume of data and the speed or flow rate of the data can be very important factors.
If you are planning to build a data pipeline for a Data Science project, then you should consider all the data sources that the ML model will need for further development. The data cleansing process is a big task for the data engineering team, which must have adequate and sufficient transformation toolsets. Data science projects work with large amounts of data, which requires storage planning. Depending on how the machine learning model will be used, either real-time or batch processing should be used.
What's next?
Big data and data growth in general pose new challenges for data architects, and data architecture requirements are constantly changing. The diversity of data, data formats and data sources is constantly increasing. In enterprise development, the value of data is obvious, and more and more processes are being automated. Increasing access to analytics and other information is required to make decisions in real time. Because of this, it is important to consider all the variables to create a scalable performance system. The data pipeline must be robust, flexible and reliable. Data quality must inspire confidence among all users. Data confidentiality is one of the most important factors in any design.
