cutting off everything unnecessary in Agile

One of the most natural development paths for a programmer is to strive to become a software architect. Designing systems is not easy, and building a complex system requires both being an expert technician and having domain expertise. It takes a lot of time and effort to develop these skills. While reading the book “A Fundamental Approach to Software Architecture” by Mark Richards and Neil Ford, I found their metaphor of “architectural katas”. In their interpretation, kata is practicing a solution to an elementary problem. I decided to try the kata approach to sharpen my system design skills.

Note: kata is Japanese martial arts and gymnastics training. This term in IT is introduced by Mark Richards in his book “A Fundamental Approach to Software Architecture”, explained on p. 99-100.

Road map

This article covers:

1. Basic approach to problem statement;
2. My approach to developing a solution;
3. My solution to the problem.

❯ Formulation of the problem

The task was set to design a platform for publishing books at the company’s request. This platform should provide the following capabilities:

1. Visitors should be able to search for and purchase books;
2. Authors should be able to upload their books for preview;
3. Company employees must become familiar with the books, manage them, and inform customers about them.

A detailed text of the kata describing specific requirements is provided.

Here

.

The “additional context” of the kata is an important block from which additional constraints can be derived and what can be logically expected from the designed system. Let’s look at everything in order:

The business came to this decision because competitors have a similar offer

This may mean that our platform must provide at least the same basic capabilities, and maintain a basic level of quality that would make the platform attractive to both authors and readers. Most of the features on it should be provided from the very beginning (in another option, some features can be added progressively, but also very quickly), because if the initial offer seems unsatisfactory, then the company simply will not be able to gain a user base. This may also mean that the platform should be integrated with existing ones so that migration to your platform is easy for both authors and readers.

There is fierce competition for authors

This means that all author-focused features must be of high quality, and the service itself must be logical, durable, and available 24/7.

This is part of a long-term strategy to modernize the publishing component of big business

From this point we can conclude that the system must be designed for scalability and extensibility, but the original solution must be integrated with existing non-new technologies or a business model in which certain processes are clearly stalling.

The information needed to publish a book (distribution, royalties, marketing) comes from several disparate systems

Again, this aspect is related to extensibility and integration. Ideally, sooner or later, all sources should be hidden under one beautiful user interface, and unnecessary ones should be eliminated altogether.

❯ How to approach the problem

There are many tools available to help you roughly design a system and outline its contours. So,

C4 model

helps visualize the system. Working with this tool involves a top-down approach: first, a maximum high-level overview is made, after which, through a gradual decomposition of components, we descend to the class level (object-oriented design). According to the concept of domain-driven design, the system must be divided along the lines of demarcation of its internal connecting areas, thereby ensuring their isolation and adaptability of the system. I think it’s important to keep the user in mind when approaching a task like this.

So I usually take this approach:

1. I identify the main categories of users and their roles;
2. I list the functional requirements (based on user stories, use cases or other similar information);
3. I outline key subject areas (in accordance with DDD);
4. I design key components in accordance with functional requirements within subject areas and covering several subject areas at once, correlating the non-functional requirements that these components have;
5. I choose technologies to implement.

So let’s practice the kata based on the above considerations!

❯ Primary users

From the statement of the problem it directly follows that the platform must support at least three user roles:

• clients
• authors
• editors/reviewers

But from the description it is also clear that we will also have:

• technical support engineers (who will help users solve problems that arise)
• order and billing managers (their job is to track book sales and manage sales, shipping, and payments)

Functional requirements

Even now, several subject areas into which the system will be divided are already clearly visible. For the sake of clarity, let’s divide the user stories into these same areas. I’ll pick a few trivial names for user stories and a few hypothetical ones that seem important to me and complement the overall picture.

Catalog:

1. The user can view a list of books;
2. The user can view details about a particular book;
3. The user can scroll through the list of books;
4. The user can search for books.

Placing orders:

1. The user can add books to the cart or remove them from there;
2. The user can view the order basket;
3. The user can place an order taking into account the status of his cart;
4. The user can choose the version of the book (electronic, paper);
5. The user can choose the address and delivery time, as well as the company that will deliver the paper book to him;
6. The user can view their orders (details and status);
7. The user can cancel orders;
8. The user can receive information about the status of their orders by email
9. The manager can view the list of orders and details about them;
10. The manager can set the order status;
11. The manager can contact the author of the order (by phone/email/chat);
12. The manager can send notifications to the user via email;
13. The manager can search for orders;
14. The manager can entrust orders to the courier (or simply transfer them to the delivery company).

Editorial work:

1. Managers can edit metadata for books;
2. Managers can publish books in the catalogue;
3. Managers can set the price of a book;
4. Managers can approve/reject chapters or entire books;
5. Managers can leave comments on chapter or book reviews;
6. Managers can notify users when new chapters or books become available;
7. The author can submit a chapter/book for review;
8. Authors can leave comments about the received review;
9. Authors can accept/reject changes made to the book during review;
10. Authors may upload chapters in file formats;
11. An author can upload an entire book as a file;
12. The author can edit the contents of the book and its metadata directly on the website;
13. Authors can view statistics about their books;
14. Authors can manage payment settings.

Considering all the listed requirements, let’s take a closer look at the subject areas they relate to.

❯ Subject areas

To distinguish between subject areas in a system, I use actors, entities, and communication patterns. If I understand these points, then it is easier for me to evaluate the quality of communication, loading, data access patterns, and data storage requirements. Below we will look at 5 subject areas, which, in my opinion, are just enough for the initial design of the system.

Catalog

• Actors: readers
• Entities: book descriptions, covers, lists, reviews, comments
• Functions: all of the above use cases

Load

Since the catalog is accessed by all users (both active clients and casual visitors), and there can be millions of such users. That is, it is assumed that the service will operate under high load. Users will receive information about the book from the catalogue, so the reading load will be particularly high.

Technical requirements

1. Availability.

We want users to be able to access the directory whenever they want. Here you can sacrifice consistency: updating the title of a book or even adding a new book may not be displayed in the catalog immediately upon the fact of this action (a delay of even a couple of seconds would definitely be acceptable here).

2. Scalability.
Since the user audience will grow (despite the fact that it is already huge), the system should easily scale and handle the amount of work proportional to the number of new users.

3. Elasticity.
If there is a sudden surge in traffic in the directory, the system must be able to quickly cope with it, and so that this process is transparent for users, and they must be able to continue working with the site as usual.

4. Failure recovery and resilience.
The catalog is probably the most important functional part of the platform provided to users. It is the catalog that new visitors first come to, and it is in this part of the platform that its main tasks are solved. If the system serving the catalog suddenly fails, it should recover very quickly, re-arranging all the useful data. Ideally, this system should not fail at all and be resilient to any exceptional conditions or network/data/hardware errors.

Business requirements

No specific business requirements are put forward here, but it is quite possible to guess which of them we will definitely have to face:

1. Content personalization
2. A/B testing support

Basket

• Actors: readers, order managers.
• Entities: items in the cart, pictures, forms (address, card credentials, user identification).
• Functions: all use cases mentioned above.

Load

The cart will be accessed by those users who are currently active customers, as well as employees who have access to orders and purchase history data. A serious load is not expected at all, since the number of buyers is much less than the total number of visitors. However, as soon as a sufficiently large amount of historical order data accumulates, the analysis of this data and operations on it can become seriously complicated. This is why it is important to separate online and offline order processing.

Technical requirements

1. Consistency.

Users will likely want all changes to their shopping cart to be visible instantly and predictably, so consistency is a top priority here.

2. Sustainability.
The shopping cart, just like the catalog, is a critical component from a user’s perspective. If user actions are unsuccessful every now and then, then users leave. Therefore, in the shopping cart, all internal and external errors must be handled carefully, and we must do everything possible to avoid losing user data.

3. Recovery after failures.
If the service suddenly fails, then you need to immediately spawn new safety instances of the basket and very quickly fill them with data so that the user can continue working. Here you can use certain security services that would inform the user about a problem that has arisen and display the most recent cached state of the recycle bin – just so that the user can make sure that all his data is safe.

4. Security.
The described cart has a page for placing an order, as well as a page for entering card details and subsequent payment. Needless to say, the service must handle this data properly.

Business requirements

Specific business requirements are not specified here either, but here are what they might be:

1. Maintain the status of the order basket between this visit and the next;
2. Recommend products on the checkout page.

Study

This component contains functions related to the author’s work.

• Actors: book authors
• Entities: documents, text
• Functions: downloading and editing chapters, sending them for review

Load

Given that there are quite a few authors, the work they do is easy to isolate and therefore distribute. The intensity of competitive processing here will be low, and the overall load is expected to be relatively low.

Technical requirements

1. Consistency and sustainability.

When editing and uploading texts, authors will likely expect this information to be securely stored.

2. Scalability.
Due to the number of books and their size, there may be performance issues, but since each author works only with his own texts, and his work is unlikely to have hundreds of huge chapters, scalability is not a particular problem here.

Business requirements

1. Notifications that the status of the manuscript has changed after review;
2. Support for “beta” versions of chapters.

Editorial CRM system

• Actors: book editors and content managers
• Entities: comments-reviews, chapters
• Functions: all of the above use cases

Load

The editorial section is similar to the author’s office: a small number of users work with this section, but the volume of data is comparable to catalog data, so search and filtering operations will take quite a long time to process.

Technical requirements

1. Interoperability.

As stated in the kata, the platform must support integration with various data sources and combine data into a single (editable) “view.” These sources come and go, so the service must be open to extensibility and further integration with the addition of new sources.

2. Consistency.
Since editors work with reviews and have access to the contents of the books themselves, any outdated data will add confusion to the process and should be discarded.

Business requirements

Based on the statement of the problem, we should be able to:

1. Notify authors that the status of the book has changed after reviewing;
2. Notify clients when new chapters are released.

Data storage system

• Actors: employee “administrators”
• Entities: orders, payments, delivered books
• Functions: all of the above use cases

Load

The storage rarely needs to be accessed directly, and there is no need to operate on huge amounts of data at a time, so no serious load is expected.

Technical requirements

1. High consistency

The data warehouse service is the source of truth for all data related to orders and books, so data integrity is of the highest priority.

2. Security.
The module must support: data masking, data restriction and other features that will protect user data from unauthorized access.

❯ How to design this service

There are many schemes of similar architecture on the Internet, each with its own advantages and disadvantages. But for this post I will use the flowchart from

this

articles on Medium. Here’s my option:

Block diagram of one of the possible solutions to this problem

Let’s break down all the components of this image.

General scheme

Of the many options available, I chose a microservices architecture as one that scales very easily and naturally provides separation of responsibilities. All services are logically derived from the above subject areas and can be deployed in Docker under k8s, in Nomad or in the Google/AWS Cloud. We are talking about the following services:

Catalog

This service will cache a minimum of data about all books that need to be displayed in the catalog, as well as specific data for each book. Also, data from recommendation or rating services will be cached here, if we have any. I recommend using couchbase as a database for efficient in-memory processing, scalability and eventual consistency. Book data is updated via notifications from Kafka (Books topic) as soon as the book is updated in the Books Service. Overall, the entire system is designed to withstand high reading loads.

Another important detail is that for security reasons, web traffic for this service is routed through a special load balancer and passes through the subnet (we don’t want clients accessing services that are not intended for them).

Orders

This service is designed in such a way that it stores all the important data about the orders placed by customers. Since consistency has the highest priority here, we will store data for this service in PostgreSQL. As soon as an order is formed, details about it are transferred to the Orders topic stored in Kafka.

Books

This service will be responsible for storing the contents of books, making this information available, distributing and delivering. When the contents of a book change, all interested parties are notified via the Books topic in Kafka, and availability information is recorded in the Books-Lock topic.

Users

Contains important information about users (clients, authors, employees) shared between different services.

Notification Center

Responsible for sending notifications to different users (clients, authors, employees) through targeted channels (email, SMS, push notifications, etc.).

Author’s content management system

This is a client API for writers, which implements the following functions: working with documents, going through the review process, viewing a profile and other logic, as well as communication with various external services. Couchbase must cache both review data and documents until the chapter or book is approved by the editorial team.

Administrator for employees

This is a client API for various employees who manage the system: editors, content managers, support engineers and others.

Thank you for reading! Tell us how you think this system could be improved or broken.

You might also want to read this: