General principles of system integration. SA for the little ones

Lyosik | Lead Systems Analyst (SA Lead)

Welcome to the block of articles for beginner systems/business analysts. Here we prepare to receive the coveted offer together

Before diving into this article, I strongly recommend that you first read the previous part about client-server architecture, which will only take 4 minutes of your time. *Link below:

In the previous article we came to the understanding that the client and server must somehow interact with each other.

And indeed, the client and server usually communicate via the Internet (although they can work in the same local network, and in general in any other types of networks). Communication occurs via such a thing as a protocol.

Protocol — is a set of rules and standards that define how data is transmitted and processed over a network.

So, the client and server interact using standard protocols, such as HTTP, FTP or lower-level ones – TCP or UDP. The protocol is usually chosen for the type of service that the servers provide.

In fact, the interaction between the client and the server is not simply via the Internet. The client contacts the server according to a specific contract (API).

Let's not be afraid of new definitions and open Google:

API Contract (Application Programming Interface) is a formal agreement that describes how different software components interact with each other. The contract defines what functions are available, how they are called, what parameters are accepted, and what data is returned.

The API sets not only the order of data transfer. It can be used to transfer a specific task (using the GET, POST, PUT, DELETE methods). We also receive the answer in a specific form (json\xml structures). *But more on that later*

Thus, we have come to the topic of this article. Yes, the above described are those very integrations. But the interaction of the client with the server is only a special case of them.

So, inhale, exhale and let's go.

The concept of integration

Integration – is the exchange of data between systems with possible subsequent processing of this data by each of the systems.

Integration can be, for example, between systems, microservices, or between the front and back.

A bit of theory

To begin, try to answer the following questions yourself:

• What is frontend and backend from your experience?

• Where to start writing a production – from the front or from the back?

Front and back

Hidden text

The frontend can be different. For example, homogeneous and heterogeneous.

And again we turn to the keeper of knowledge:

Homogeneous frontend — is a situation where all components and technologies in a project use the same standards and approaches. For example, using one framework (for example, React or Vue) for all parts of the application.

Heterogeneous frontend — this is when a project uses different technologies, frameworks or libraries. For example, part of the application can be written in React, and another part in Angular or Vue.

Can the back end also be heterogeneous? Quite possibly. Especially when it comes to microservices.

And since we’re talking about microservices, let’s dwell on this topic a little.

Monolith and microservices

Monolith — is a hierarchical architecture. Each application layer is responsible for its part of the functionality. All software components of the system are interdependent, due to the use of built-in data exchange mechanisms between systems.

Advantages: ease of E2E testing, ease of deployment.

Microservices — is a symmetrical architecture. Each service has its own base and is responsible for a business function.

Advantages: stack independence, scalability, ease of unit testing.

Lifehack: no one will guess that a monolith is a monolith if you don't allow people to access the code base!:)

Important!

I would like to emphasize that the above concepts of Monolith and Microservices are applicable exclusively within the framework of client-server architecture! That is, if you are involved in developing a standalone application that performs all calculations on the client's machine (for example, some single-user calculator), then you should not call it a monolith.

Let's move on.

Back office

And we introduce another purely logical concept, such as Back-office. Roughly speaking, if we take a store selling something, that is, a web part where customers go (for example, ozone), and inside ozone there are other applications (in which orders are tracked, accounting, etc.), then all this together is Back-office.

Back office — is the unification of all fronts where the customer's employees or clients work

Let's look at the topic of integrations from a slightly different angle.

Interaction from a data flow perspective

Data flow — is the transmission and reception of one or another information. Data flow displays the direction and the data itself, which moves between external entities and data storages using processes.

Types of data streams:

• Introductory – information is received by the application, after which it is read;

• Conclusions – the program transmits data with subsequent recording into streams.

Here, two new concepts immediately arise before us – Producer and Consumer.

Producer – is the one who produces/sends data.

Consumer – this is the one who receives the data.

In the diagram below we see the following: The user fills out a form and sends the data. IT System 1 accepts this data and saves it to the database.

Try to answer the question: who is the Producer and who is the Consumer in this diagram?

Let's add two more concepts to our IT vocabulary:

Transform – is the one who changes the data.

Storage – this is the one who saves the data.

Important!

If we, as users, not only upload data into a system, but also then work with that data, then we also become a Consumer in relation to that system.

Next, we will introduce such concepts as Statefull and Stateless systems.

Stateful — the system stores information about previous states or sessions.

Stateless — the system does not save information about previous states or sessions.

And knowing this, we move on to another concept – the data bus.

Data bus — is a middleware that provides message conversion into the required format, transaction control, routing taking into account the meaning, even distribution of the load on services and security of data exchange.

The data bus is a special case of Stateless systems. But it still violates the Stateless principle itself, because it stores data until it is processed.

By default, the bus operates on the FIFO principle.

Yes, yes, something new again. Let's remember:

FIFO And FILO — are abbreviations that describe two different management methods data queues.

FIFO (First In, First Out) — is a method in which the first element added to a queue is the first element to be retrieved. The queue operates on a first-in, first-out basis. An example of FIFO is a store queue: the first customers to arrive at the queue are the first to be served.

FILO (First In, Last Out) — is a method in which the first element added to a data structure is the last one to be retrieved. A queue operates on a first-in, last-out basis. An example of FILO is a stack: the last element added is the first one to be retrieved.

In addition to the regular data bus, there is also the so-called Native Kafka bus.

Kafka — is a service that allows for real-time, high-bandwidth messaging between different systems. Kafka is a cluster of several brokers, each of which services its own portion of the overall load.

The Native Kafka bus is unique in that it violates the FIFO principle and is not Stateless.

Since Kafka stores data longer and systems interacting with it may request older messages from it, it is necessary to specify the order in the message structure. For example:
1. Formation of an order;
2.Payment;
3. Delivery.
This way, when the system receives a delivery message, it will know that it also needs to wait for the order and payment messages.

In general, message brokers are a very large topic, deserving a separate article. But now, at first, it is enough for you to just know that such things also exist.

Well, we've reached the finish line.

Types of integrations

1. File sharing. Data exchange via files (e.g. CSV, XML) that are downloaded or uploaded.

2. Base to base (or common database). Direct interaction between two or more systems through a common database.

3. Using HTTP/HTTPS. Data exchange between different systems and applications via standard data transfer protocols on the Internet.

4. Data buses. Data exchange via a centralized platform.

5. API. Data exchange via standardized software interfaces.

6. Push channels. Real-time exchange of data and events, where one system “pushes” information to another system, without the need to request the information.

7. Using Native Libraries. Embedding ready-made software components directly into the application code, which allows the application to use the functionality of other systems or services without the need for direct interaction with their API.

Try to determine for yourself which of these types use synchronous interaction methods and which use asynchronous ones?

Let's take a closer look at using HTTP/HTTPS and APIs.

HTTP and HTTPS

At the beginning of this article we said that the interaction between the client and the server occurs according to a protocol.

Let me remind you,

Protocol — is a set of rules and standards that define how data is transmitted and processed over a network.

Some of the most popular protocols are HTTP and HTTPS:

HTTP (HyperText Transfer Protocol) — Hypertext Transfer Protocol, an application-level network protocol that was originally intended to receive hypertext documents in HTML format from servers, and over time has become a universal means of interaction between nodes of both the World Wide Web and isolated web infrastructures.

HTTPS (HyperText Transfer Protocol Secure) — an extension of the HTTP protocol to support encryption for increased security.

Alongside the topic of the HTTP protocol are the so-called response statuses.

HTTP Response Statuses — are codes that the server sends to the client (usually a web browser) in response to an HTTP request. They help determine the result of processing the request.

Everything would be fine, but the HTTP and HTTPS protocols have two main disadvantages:

• The connection between systems lasts only as long as the communication session is in progress. That is, one system makes a request to another, a connection is established. Until the second system responds, the first waits (the connection is maintained). And only after the second system responds, the connection is broken;

• With http-request, the layout is returned in the response. That is, we depend on formatting, tags, styles. Accordingly, if the data provider changes the layout, everything will break.

API

An API (Application programming interface) is a contract that a program provides. “You can address me in this way and in that way, and I agree to do this and that.”

The API includes:

• the operation itself that we can perform;

• data that comes into the input;

• the data that is output (data content or error message).

There are different data exchange formats used in APIs. We will briefly touch on two of them now:

<person>
  <name>John Doe</name>
  <age>35</age>
  <email>john.doe@example.com</email>
</person>

 {
   "name": "John Doe", 
   "age": 35, 
   "email": "john.doe@example.com"
 }

And remember: any API is based on the http protocol, i.e. the connection is always maintained.

Respectively,

API – this is a special http-request-response, which transmits data without layout in XML or JSON format.

In general, this is all I wanted to discuss in this article. Of course, now we have only superficially “touched” the integrations, but I think that at least a basic understanding of “what's what” has formed in you 🙂

In my TG channel I have collected a small selection of useful materials that will help to consolidate and increase knowledge on client-server architecture and principles of integration. Follow the link below, subscribe and use it to your health 🙂