[HTTP API & REST] Advantages and disadvantages of the HTTP API

These are chapters 36 of the “HTTP API & REST” section of my API books. The second edition of the book will contain three new sections: API Patterns, HTTP APIs and REST, SDKs and UI Libraries. If this work was useful to you, please rate the book on GitHub, Amazon or GoodReads. English version on Substack.

After three introductory chapters with clarification of the main terms and concepts (such, alas, is the price of the popularity of technology), the reader may have a reasonable question – why does such a dichotomy exist at all: some APIs rely on standard HTTP semantics, and some completely from it abandoned in favor of newly invented standards. For example, if we look at response format in JSON-RPC, we will find that it could easily be replaced by standard HTTP protocol facilities. Instead of

HTTP/1.1 200 OK

{
  "jsonrpc": "2.0",
  "id",
  "error": {
    "code": -32600,
    "message": "Invalid request"
  }
}

the server could just answer 400 Bad Request (with passing the request ID, let’s say, in the header X-OurCoffeeAPI-Request-Id). However, the developers of the protocol saw fit to develop their own format.

This situation (not only specifically with JSON-RPC, but with almost all high-level protocols on top of HTTP) has developed for many reasons, including various historical ones (for example, the inability to use many HTTP features from early implementations XMLHttpRequest in browsers). However, new variants of RPC protocols that use the absolute minimum of HTTP capabilities continue to appear today.

To resolve this paradox, let’s look at one fundamental difference between using application layer protocols (as in our JSON-RPC example) and pure HTTP: if an error 400 Bad Request is transparent to almost any network agent, then an error in the native JSON-RPC format is not – firstly, because only an agent with JSON-RPC support can understand it, and, secondly, and more importantly, in JSON- RPC request status is not meta information. The HTTP protocol allows you to read details such as the method and URL of the request, the status of the operation, the request and response headers, without reading the entire request body. For most higher-level protocols, including JSON-RPC, this is not the case: even if the agent has support for the protocol, it still needs to read and parse the response body.

How can this very ability to read metadata be useful to us? The modern client-server interaction stack is (as Fielding predicted) multi-layered. We can distinguish a set of agents of different levels, which, one way or another, process network requests and responses:

the developer writes code on top of some framework that sends requests;
the framework is based on the programming language API, the compiler or interpreter of which, in turn, relies on the operating system API;
the request reaches the server, possibly through intermediate HTTP proxies;
the server, in turn, also represents several layers of abstraction in the form of a framework, programming language and OS;
in front of the destination server, as a rule, there is a web server that proxies the request, and often more than one;
in modern cloud architectures, an HTTP request will go through several abstractions in the form of proxies and gateways before reaching the final handler.

The main advantage of following the letter of the HTTP standard is the ability to rely on the fact that intermediate agents, from client frameworks to API gateways, are able to read the request metadata and perform some actions using it – configure the requery policy and timeouts, log, cache , shard, proxy, and so on – without the need to write any additional code.

If we try to formulate the main principle of HTTP API development, we will get something like this: it would be better if you designed the API so that intermediate agents could read and interpret request and response meta-information.

Unlike most alternative technologies, the main impetus for the development of which comes from one large IT company (Facebook, Google, Apache Software Foundation), HTTP API tools are developed by many different companies and collaborations. Accordingly, instead of a homogeneous but limited framework, for the HTTP API we have many different tools, such as:

various proxies and API gateways (nginx, Envoy);
various specification description formats (first of all, OpenAPI) and related tools for working with specifications (Redoc, Swagger UI) and code generation;
Software for developers that allows you to conveniently develop and debug API clients (Postman, Insomnia), and so on.

Of course, most of these tools are also applicable to working with APIs that implement alternative paradigms. However, it is the ability of intermediate agents to read HTTP request metadata that makes it easy to build complex pipelines such as exporting nginx access logs to Prometheus and getting convenient monitoring of response status codes in Grafana out of the box.

Separately, we note that the HTTP API is today the default choice when developing public APIs. For the reasons stated above, no matter how the partner’s technology stack is arranged, he will be able to integrate with the HTTP API without much effort. At the same time, the prevalence of technology lowers both the entry threshold and the qualification requirements for partner engineers.

The main disadvantage of the HTTP API is that intermediate agents, from client frameworks to API gateways, are able to read request metadata and perform some actions using them – set up re-request policy and timeouts, log, cache, shard, proxy, and so on – even if you didn’t ask them to. Moreover, since the HTTP standards are complex, the concept of REST is unclear, and software developers are not ideal, intermediate agents (and partner developers!) Can interpret request metadata wrong. This is especially true for some exotic and difficult to implement standards. As a rule, one of the reasons for the development of new RPC frameworks is the desire to ensure the simplicity and consistency of working with the protocol, in order to reduce the field for potential errors in the implementation of API integration.

Performance Issues

In favor of many modern alternatives to the HTTP API – such as GraphQL, gRPC, Apache Thrift – an argument is often made about the poor performance of the JSON-over-HTTP API compared to the technology in question; More specifically, the following problems are called:

Format redundancy:
- in JSON, it is necessary to transfer the names of all fields every time, even if an array of a large number of identical objects is transferred;
- a large number of technical characters – quotes, brackets, commas – and the need to escape service characters in strings.
The HTTP API, in response to a request for a resource, returns a representation of the resource as a whole, although only individual fields may be of interest to the client.
Slow performance of data serialization and deserialization operations.
The transfer of binary data requires additional encoding, for example, through Base64.
Poor performance of the HTTP protocol itself (in particular, the inability to multiplex multiple requests and responses over a single connection).

Let’s be honest here: most of the existing implementations of the HTTP API do suffer from these problems. However, we take the liberty of stating that all of these problems are mostly contrived and not given much attention because these overheads are not noticeable to most API vendors. In particular:

If we are talking about format redundancy, then an important caveat must be made: all of the above is true if we do not apply compression. Comparisons showthat the use of gzip practically eliminates the difference in the size of JSON documents relative to alternative binary formats (and there are also archivers specially designed for text data, for example, brotli).
Generally speaking, if such a need arises, then within the framework of the HTTP API it is quite possible to regulate the list of returned response fields, this is quite consistent with the spirit and letter of the standard. However, we should note that the traffic savings on returning partial states (which we discussed in detail in the Partial Updates chapter) are very rarely justified.
If you use standard JSON deserializers, the difference compared to binary formats can be really big. If, however, this overhead is a problem, it is worth turning to alternative deserializers – in particular, simdjson. Thanks to the optimized low-level code, simdjson shows excellent performance, which can only be lacking in very exotic APIs.
Generally speaking, the HTTP API paradigm implies that separate endpoints are provided for binary data (such as images or video files). Passing binary data in the JSON response body is only necessary in cases where a separate request behind them is a performance problem. Such a problem does not actually exist in server-2-server communication and in HTTP 2.0 and above.
HTTP 1.1 is indeed not ideal in terms of request multiplexing. However, alternative API organization paradigms for solving this problem rely on…HTTP version 2.0. Of course, the HTTP API can also be built on top of version 2.0.

Just in case, let’s clarify again: JSON-over-HTTP API really loses in terms of performance to modern binary protocols. However, we take the liberty of asserting that the performance of a well-designed and optimized HTTP API is sufficient for the vast majority of subject areas, and the real gain from switching to alternative protocols will be insignificant.

Advantages and disadvantages of the JSON format

As you can see, most of the claims against the HTTP API concept are not related to HTTP at all, but to the use of the JSON format. Indeed, nothing prevents you from developing an API that will use any binary format instead of JSON (including the same Protocol Buffers), and then the difference between the Protobuf-over-HTTP API and gRPC will be reduced only to the (not) use of verbose URLs, status codes, and request and response headers (and the resulting (not) ability to use one or another standard software out of the box “).

However, in many cases (including this book), developers prefer text JSON over binary Protobuf (Flatbuffers, Thrift, Avro, etc.) for a very simple reason: JSON is very easy and convenient to read. Firstly, it is textual and does not require additional decryption; second, the field names are included in the file itself. If a protobuf message cannot be read without .proto-file, then you can almost always try to understand from the JSON document what kind of data is described in it. Together with the fact that when developing the HTTP API, we also try to follow the standard semantics of the rest of the binding, as a result, we get an API whose requests and responses (at least in theory) are easy to read and intuitive.

In addition to human readability, JSON has another important advantage: it is as formal as possible. It does not contain any constructions that can be interpreted differently in different architectures (up to restrictions on the lengths of numbers and strings), and at the same time it conveniently fits into native data structures (index and associative arrays) of almost any programming language. From this point of view, we actually had no other choice as to what other data format we could use when writing the code examples for this book.

In the case of developing an API of less general purpose, we recommend approaching the choice of format according to the same principle:

weigh the overhead of preparing and implementing tools for reading binary formats and decrypting binary protocols against the overhead of non-optimal data transfer;
evaluate whether you need a high-quality, but limited set of software that comes with an alternative format, or whether you prefer the ability to use a wide range of tools that can work with the HTTP API, even if they are not always of high quality;
estimate how much it will cost you developers who can work with this format.