OpenAPI Comparator – detector of changes that break [большие] API

Protecting yourself from API changes that could affect your clients is one of the biggest challenges when developing an HTTP API. If you make breaking changes, you create serious problems for customers. A stable API is the key to customer satisfaction and ultimately business success.

But there is one catch: customer needs change, which often requires the development of the API itself. For example, adding parameters to work with a given resource or removing other parameters to simplify the entire API.

Preventing breaking changes becomes more difficult as the API grows. With over 250 API endpoints, we’ve reached the point where manual breaking change detection has become untenable. That’s why we were looking for an automated solution.

What is breaking change?

In a nutshell, a breaking change is a change to an API that has the potential to cause crashes in applications that use the API.

Some breaking changes are easier to identify than others. For example, deleting an endpoint is considered a breaking change in most cases because it breaks client integration. But some other changes can be harder to catch, such as changing an optional value in an enum or query parameter to a required value.

Not all changes are critical. Adding a property or optional parameter to the schema shouldn’t make any difference, as applications using the API don’t need to be updated to run.

Other changes may not be reflected in the API contract, such as behavior changes. These breaking changes are harder to detect and will not be covered in this article.

Determining how critical a change is is mostly API dependent. An early-stage API that doesn’t prioritize stability may have a different definition of a breaking change than a long-standing API. Link incomplete list supported changes in Criteo.

To write API contracts, we use OpenAPI specification standard (OAS). It is by far the most advanced and supported standard for describing the HTTP API. You may have heard of it called Swagger Specification.

Breaking Change Examples

Let’s take a look at updating an enum value and see what it looks like in the OpenAPI specification. The starting point will be the following query parameter definition animal-typewhose values ​​are cat, dog or panda:

parameters:
  - in: query
    name: animal-type
    schema:
      type: string
      enum: [
        cat,
        dog,
        panda
      ]

Let’s say the API is updated and the request parameter animal-type no longer takes on a value panda. The application might expect this value to still be valid, but the API has changed so panda no longer accepted:

{
    "traceId": "56ed4096-f96a-4944-8881-05468efe0ec9",
    "type": "validation",
    "code": "validation-error",
    "instance": "/clinics/1234/pets",
    "title": "Unknown enum value",
    "detail": "Unexpected enum value: panda"
}

Change detection, but where?

Criteo API complies with the industry standard; it is divided into subservices, each of which is responsible for a part of the API. The API Gateway – as a single entry point for the API – is placed in front of these services. Each API call first reaches the gateway, which forwards the call to the correct underlying service.

Through the routing configuration in the service registry, the gateway knows which service to route the call to.

And if you want to know more about how our API design has evolved, read this article.

The service registry is responsible for building the routing configuration. Each internal service has its own routing configuration and OpenAPI specification contract. The configuration and contract are combined and provided by the registry. This seems to be a good place to spot breaking changes.

How?

The routing configuration and OpenAPI specification of each endpoint is stored in a git repository. This allows you to check for contract changes and easily view history.

In addition, a configuration change request triggers automatic checks to make sure the routes and OpenAPI specifications are correct. With this process, each update to the external API is versioned, automatically validated, and reviewed by a human.

One of the automatic checks ensures that no breaking changes have been made. The check compares the aggregated OpenAPI specification in the repository with the working specification. Each difference found is classified. Some types of classification are considered as a breakdown, others are not. If differences are found, and at least one of them we consider critical, then the pull request is rejected.

The ability to effectively compare OpenAPI specifications is a key component of the system, as shown above, and this is where OpenAPI comparator.

OpenAPI Comparator

Most of our stack is written in .NET, so we were looking for a C# library capable of comparing OpenAPI specifications. Then attention was drawn to one open source project: openapi-diff, developed by Microsoft Azure. Unfortunately it was not available as a NuGet package. And, more importantly, openapi-diff only works with OpenAPI 2.0, which was replaced by OpenAPI 3.0 in 2017. We decided to create an internal fork of the project and perform a migration in order to compare the Open API specification version 3.0.

Now that the migration is complete, it’s fair to share the tool with the community: OpenAPI Comparator, a C# library that compares OpenAPI version 3.0 specifications. Our library is available as NuGet package. The library can detect many kinds of differences, which is useful when comparing two APIs or two versions of the same API. Additional metadata is returned for each difference found:

• An explicit description of the change.
• Its location in the old and new specifications.
• Link to documentation about the broken comparison rule.
• Type: remove, update or add.
• The severity of the detected change can be as follows: info, Warning or error.

The comparator has a pretty simple API. The code below corresponds to one method that uses two OpenAPI specifications for comparison as arguments:

IEnumerable<ComparisonMessage> OpenApiComparator.Compare(
    string oldOpenApiSpec, 
    string newOpenApiSpec);

The tool also has a command line interface, this interface is available as NuGet package:

openapi-compare --old path/to/old-spec.json --new path/to/new-spec.json

From the list of detected changes from the comparator, you can easily understand whether breaking changes have been made.

Conclusion

When a client starts using an API, there is an implicit agreement that the same version of the API should not be updated critically. Protecting APIs from breaking changes is a strategic investment and should not be underestimated.

But this protection can be tedious and error-prone, and as the API grows, the problem will get worse. That’s why we designed and implemented an automated review process that compares a potential update to a released API. If a breaking change is detected, the update is rejected.

Automating the detection of breaking changes has resulted in significant performance and efficiency gains. And we are excited to be able to help the large community of API developers by open source OpenAPI comparator.

A useful theory and even more practice with immersion in the IT environment are waiting for you in our courses: