Samoyed CMG – API Generator

I wrote an earlier article Generating a website API based on user-specified functions, but the information there was about the final implementation (besides theoretical), and, as expected, no one understood why this was needed at all. Therefore, I will try to paint it from the other side: from the task to its solution through the API generated in Samoyed CMG.

Task description

Suppose we have a small site with a list of articles with pagination. Articles are written by site users.
On the main page we display a list of the last 10 articles. The title and author are listed. When you click on the title, the page with the selected article is displayed. The article page displays the title, content + author.
The simplest database table schema is shown below.

Implementation

The site has two routes:

To implement the above, the following functions are required:

That’s the whole backend implementation. View functions get return the data that we pass in the controllers to the templates and generate a page for the client on the server. Functions themselves separate logic from presentation. View functions put change the data. This approach will allow, if necessary, to change the logic of these functions without rewriting the code that uses them.
Due to simplicity, I will not give the implementation of the functions, since this is just one SQL query per function.

Implementation with caching

The implementation described above will work fine with a small number of users on the site. However, if their number grows (and this is the task of any site), then problems will begin due to the maximum number of connections to the database. Those. if suddenly 100 people open the site at the same time, then with the maximum number of connections = 20, 80 of these 100 will receive a database connection error. After that, they will refresh the page and again 60 out of 80 will see an error. Such errors scare away users, which means they should be avoided. And CACHETING will help us.

The simplest caching option is caching by time (LifeTime). Those. Specify the lifetime of the data. The function returns data and writes it to CACHE for N seconds. The name of the function + its parameters acts as the KESH key. At the next request for the same data, they are searched in the CASH + checking the lifetime, and if the time has not expired, then the data is taken from the CASH.

A more complicated version of working with the CACHE is clearing the CACHE when changing the data that is in the CACHE. For example, we call the function articleGet which, when called, generates a key articleGet.$id and checks for the presence of a value by key in the CASH. If it is not there, then we make a selection from the database, if it is, then we return the value from the CASH. We also change the function articlePutin which we also generate the key articleGet.$id and delete the value from the CASH by this key. This means that if after that the articleGet function is called with the number of the article that was changed, it will be re-selected from the database.

The timed caching option works well for the article list page, as it will change quite often (because articles are added). And for the functions of obtaining articles and users, a more complex option is suitable with deleting data from the CASH (because articles and data will change quite rarely).

As a result, the scheme will be the following


On the scheme, different types of functions are highlighted in color + functions of the form puton which functions of the form get (in the previous diagram of the function put were not specified, since they were “on their own” and there was no dependence). These dependencies are important, because when changes are made, values ​​must be removed from the cache.

In total, functions can be of the following types:

N+1 request problem

The N + 1 problem occurs when the data access framework executes N additional SQL queries to retrieve the same data that can be retrieved with a single SQL query.
As an example, we can take the function articleList. When implemented “on the forehead” it selects a list of article identifiers (1 request), and then selects information about each article by its identifier (N requests). In practice, two requests can be dispensed with. First, select a list of articles, and then generate a list of user identifiers and, with a second request, select information about all users at once.
However, the variant with two requests breaks the caching system, since for the correct CACHE function articleGet must be called separately for each identifier.
And here it will be useful grouping function calls.

Grouping function calls

An API function with call grouping does not return a result, but a promise to return a result later. Those. the function will run in asynchronous mode. At the same time, the parameters of not one call, but several at once will come to the function. Which allows us to successfully solve the problem of N + 1 requests.

To understand the source code

`ticleList` function

// Эта анонимная функция определяет список параметров функции API
// Параметр с именем return указывает тип возвращаемого значения
return function (int $page, int $perPage, array $return): \Closure {
    // Эта анонимная функция определяет функцию, которая выполняет группировку вызовов всех функций API
    // Параметр $fnCalls опреляет тип функции Calls[Get|Put|Lifetime|Direct]
    // $fnCalls является итерируемым объектом и позволяет перебирать все вызовы
    return function (CallsLifetime $fnCalls, IDatabase $database): void {
        // Создаём объект для кеширования вызовов в памяти
        $cacheCall = new CacheCall();
        // Перебираем все вызовы
        foreach ($fnCalls as $fnCall) {
            // Установить время жизни для вызова функции $fnCall КЕШ-а = 5 минут
            $fnCall->setTtl(5 * 60);
            // Выбрать список идентификаторов статей по входным параметрам функции API: page и perPage (и завешировать результат в памяти)
            $articles = $cacheCall($fnCall->arg('page'), $fnCall->arg('perPage'), function (int $page, int $perPage) use ($database) {
                return $database->select('articles', 'id')->orderBy('id', false)->pagination($perPage, $page);
            });
            // Для каждого идентификатора статьи вызвать функцию получения статьи
            $waits = [];
            foreach ($articles->items() as $article) {
                $waits[] = $fnCall->fns()->articleGet($article['id']);
            }
            // Ждать завершения выполнения всех функций
            $fnCalls->all($waits)->wait(function ($items, $err) use ($fnCall, $articles) {
                // Вернуть результат (точнее установить результат)
                $fnCall->setValue($articles->args($items));
            });
        }
    };
};
`ticleGet` function

// Эта анонимная функция определяет список параметров функции API
// Параметр с именем return указывает тип возвращаемого значения
return function (int $id, array $return): \Closure {
    // Эта анонимная функция определяет функцию, которая выполняет группировку вызовов всех функций API
    // Параметр $fnCalls опреляет тип функции Calls[Get|Put|Lifetime|Direct]
    // $fnCalls является итерируемым объектом и позволяет перебирать все вызовы
    return function (CallsGet $fnCalls, IDatabase $database): void {
        // Выбрать информацию о статьях
        // $fnCalls->args() - получает список параметров всех вызовов
        // ArrArr::value('id', $fnCalls->args()) - получает уникальные значения параметра id
        // ArrArr::groupBy('id', ...) - группировать массив по полю id
        $articles = ArrArr::groupBy('id', $database->select('articles')->in('id', ArrArr::value('id', $fnCalls->args()))->get());
        // Разобрать выбранные данные по вызовам API функций
        foreach ($fnCalls as $fnCall) {
            // Информция о статье
            $article = $articles[$fnCall->arg('id')];
            // Указать зависимость вызова от функции PUT
            // Т.е. при вызове функции articlePut с указанным параметром КЕШ функции $fnCall будет сброшен
            $fnCall->fns()->articlePut($article['id']);
            // Выбрать информацию о пользователе
            $fnCall->fns()->userGet($article['author_id'])->wait(function ($value, $err) use ($article, $fnCall) {
                // Информация о пользователе
                $article['author'] = $value;
                // Установить значение функции
                $fnCall->setValue($article);
            });
        }
    };
};
`userGet` function

// Эта анонимная функция определяет список параметров функции API
// Параметр с именем return указывает тип возвращаемого значения
return function (int $id, array $return): \Closure {
    // Эта анонимная функция определяет функцию, которая выполняет группировку вызовов всех функций API
    // Параметр $fnCalls опреляет тип функции Calls[Get|Put|Lifetime|Direct]
    // $fnCalls является итерируемым объектом и позволяет перебирать все вызовы
    return function (CallsGet $fnCalls, IDatabase $database): void {
        // Выбрать список уникальных идентификаторов пользователей
        $author_ids = ArrArr::value('id', $fnCalls->args());
        // Выбрать информацию о пользователях
        $users = ArrArr::groupBy('id', $database->select('users')->in('id', $author_ids)->get());
        // Разобрать выбранные данные по вызовам
        foreach ($fnCalls as $fnCall) {
            // Информация о пользователе с идентификатором вызова $fnCall->arg('id')
            $user = $users[$fnCall->arg('id')];
            // Указать зависимость вызова от функции PUT
            $fnCall->fns()->userPut($user['id']);
            // Установить значение функции
            $fnCall->setValue($user);
        }
    };
};

Grouping function calls allows you to use caching. Those. only those function calls for which there are no values ​​in the CASH will get into the group call function. A change in the value in the CASH will occur only when the corresponding data change function is called.

The API service allows you to call asynchronous functions in synchronous code:

  // Вызов функций API
  $articles = $api(function (CallAll $apiCall) use ($request) {
      // Вызватиь функцию articleList
      $apiCall->fns()->articleList($request->query()->get('p', 0), 10)->wait(function ($value, $err) use ($apiCall) {
          // Установить результат вызова API
          $apiCall->setValue($value);
      });
  });

As a result, in $articles will be the result that was set using the call $apiCall->setValue($value);. Those. data from the asynchronous block is returned to the synchronous code.

It is convenient to implement business logic functions in the API. In this case, the developer will not need to think about the implementation of caching, it all comes out of the box. It is enough just to write an API function of a given type. When working with a database, in my opinion, a very useful “trick”. The example above clearly shows the reduction in the number of samples from the database. After the first call, only the articleList() function will actually be called, and even then not always, but only when the cache lifetime expires. Those. once every 5 minutes.
Functions are quite easy to use not only on the backend, but also on the frontend. To do this, it is enough to automatically generate an access point convenient for your task. Since there is a list of all functions and their parameters, this is not a difficult task. This means that we can call the same function, for example, aticleList(), both on the server and on the client. The returned data will be identical. Those. according to the same function, you can generate a page both on the server (for example, php + twig) and on the client (for example, vue / react / angular).

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *