Conflict-free merging of any number of APIs

Namespace is an important concept in programming that allows you to group elements and prevent name conflicts. In this post, we'll show how we apply this concept to APIs to facilitate the composition and integration of different services.

We'll show you how to integrate 8 services: SpaceX GraphQL, 4x GraphQL using Apollo Federation, REST API using OpenAPI Specification, PostgreSQL based API and Planetscale-Vitess (MySQL) based API in just a few lines of code, fully automatic, without any hassle. – or conflicts.

When you install the package npm, it is in its own namespace. One such package is axiosa very popular client for making HTTP requests.

to install axiosyou run the following command:

yarn add axios

This installs the dependency axios to your folder node_modules and adds it to your file package.json.

You can now import and use the code provided by the package axiosin the following way:

import axios from "axios";
const res = await axios.get("https://example.com");

Import the dependency, give it a name, in this case just axios, then use it. We could also rename axios V bxios. Renaming imports is an important element of dependency management to avoid conflicts.

One of the basic rules is that you cannot have two imports with the same name, otherwise you will have a name conflict and it will not be clear how the program should be executed.

Should we run axios or bxios?

Okay, enough introduction. You're probably already familiar with all this, what does it have to do with APIs?

Much! At least that's what I think. This whole workflow is amazing!

You can write code, package it as a package npm, publish it, and others can import and use it very easily. It's such a fun way to collaborate through code.

What does this look like when using the API? Well, it's not such a well-oiled machine. With APIs we are still in the stone age when it comes to this workflow.

Some companies offer an SDK that you can download and integrate. Others simply publish a REST or GraphQL API. Some of them have the OpenAPI specification, others simply offer their own custom API documentation.

Imagine if you had to integrate 8 services to get data from them. Why don't you just run something like yarn add axios and get the job done? Why is it so difficult to combine services?

Problem – how to combine APIs without conflicts

To achieve this, we need to solve a number of problems.

1. We need to come to a common language, a universal language, to unify all our APIs

2. We need to find a way to “spatially” separate our APIs to resolve conflicts

3. We need a runtime to perform “spatially separated” operations

Let's look at the problems one by one.

GraphQL: A Universal API Integration Language

The first problem to solve is that we need a common language on which to base our implementation approach. Without getting too carried away, let me explain why GraphQL is great for this purpose.

GraphQL has two very powerful features that are essential for our use case. On the one hand, it allows us to request exactly the data we need. This is very important when we use many data sources, as we can easily drill into the fields that interest us.

On the other hand, GraphQL allows us to easily build and follow links between types. For example, you might have two REST endpoints, one with messages and one with comments. With the GraphQL API in front of them, you can build a link between two objects and allow your users to retrieve messages and comments with a single request.

Additionally, GraphQL has a thriving community, with many conferences and people actively participating, building tools around the query language, and much more.

GraphQL and Microservices: Schema Stitching vs. Federation

It's worth noting that GraphQL also has a weak point when it comes to API integration. It has no concept of namespaces, which makes using it for API integration a bit difficult, until now!

When it comes to service integration, there are currently two main approaches to solving the problem. First is Schema Stitching, and then Federation.

With Schema Stitching, you can stitch together GraphQL services that don't know about stitching. API merging happens in a centralized location, the GraphQL API Gateway, without the knowledge of the services.

The Federation defined by Apollo, on the other hand, offers a different approach. Instead of centralizing the stitching logic and rules, federation distributes them across all GraphQL microservices, also known as Subgraphs. Each Subgraph defines how it contributes to the overall scheme, fully aware that other Subgraphs exist.

There is no “best” solution here. Both approaches are good for microservices. They're just different. One prefers centralized logic, while the other suggests a decentralized approach. Both approaches have their own problems.

However, the problem of service integration goes far beyond federation and schema stitching.

One count for everyone, or not!

Basic template Principled GraphQL concerns integrity and states:

Your company should have one unified graph, not multiple graphs created by each team. By having one graph, you maximize the value of GraphQL:

• More data and services can be obtained from one request

• Code, requests, skills and experience are transferable between teams

• One central directory of all available data that all graph users can look at

• Implementation costs are minimized because the work to implement the graph is not duplicated

• Centralized graph management – such as uniform access control policies – becomes possible

When teams create their own individual graphs without coordinating their work, it inevitably causes their graphs to begin to overlap, adding the same data to the graph in incompatible ways. At best, this is costly; at worst, it creates chaos. This principle must be followed as early as possible in the company's adoption of the graph.

Let's compare this principle with what we learned about the code above, you know, the example with axios And bxios.

More data and services can be obtained from one request

Imagine there was one giant package npm for a company with all its dependencies. If you would like to add axios in your package npm, you would have to manually copy all the code into your own library and make it “your” package. This would not be supported.

One single graph sounds great when you're in complete isolation. In reality, however, this means that you should add all external APIs, all “packages” that you don't control, into your one graph. You must maintain this integration yourself.

Code, requests, skills and experience are transferable between teams

This is true. With just one graph, we can easily share requests between teams. But is this really a function? If we separate our code into packages and publish them separately, it will be easy for others to choose exactly what they need.

Imagine one graph with millions of fields. Is this truly a scalable solution? How about just picking the subparts of the giant GraphQL schema that really matter to you?

One central directory of all available data that all graph users can look at

With just one schema we can have a centralized directory, that's true. But remember that this directory can only represent our own API. What about all the other APIs in the world?

Also, why can't we have a directory of multiple APIs? Just like packages npmwhich you can search and view.

Implementation costs are minimized because the work to implement the graph is not duplicated

I would argue that the opposite is actually true. Especially with Federation, Apollo's solution for implementing a graph, it becomes much more difficult to maintain your graph. If you want to deprecate type definitions in multiple subgraphs, you will have to carefully coordinate the change across all of them.

Microservices are not truly micro if there are dependencies between them. This pattern is rather called a distributed monolith.

Centralized graph management – such as uniform access control policies – becomes possible

It's interesting what should be possible but isn't. We have not yet seen a centralized access control policy system that adds role-based access control to federated graphs. Oh actually it is one of our featuresbut let's not talk about security today.

Why does the One Count principle make no sense?

Building one single graph sounds like a great idea when you're completely isolated on a small island with no internet. You probably don't intend to use or integrate third party APIs.

Anyone else who is connected to the internet will probably want to integrate external APIs. Want to check your sales using the Stripe API? Send emails via Mailchimp or Sendgrid? Do you really want to manually add these external services to your One Graph?

The One Count principle does not pass the reality test. What we need instead is a simple way to make multiple graphs!

The world is diverse. Many great companies offer really good products through APIs. Let's make it easier to build integrations without having to manually add them to our One Graph.

GraphQL Namespaces: Merging Any Number of APIs Without Conflicts

This brings us to the second problem, name conflicts.

Imagine that both Stripe and Mailchimp define a Customer type, but each has a different definition of Customer, with different fields and types.

How can both types of Customers coexist in the same GraphQL schema? As suggested above, we borrow a concept from programming languages, namespaces!

How to achieve this? Let's look at this problem a little. Since GraphQL doesn't have a built-in namespaces feature, we'll have to get a little creative.

First, we need to resolve any naming conflicts for types. This can be done by adding a suffix to each “Customer” type with a namespace. So, we would have “Customer_stripe” and “Customer_mailchimp”. First problem solved!

Another problem we might encounter is field name conflicts in the root operation types, i.e. Query, Mutation and Subscription types. We can solve this problem by adding a prefix to all fields, for example, stripe_customer(by: ID!) And mailchimp_customer(by: ID!).

Finally, we need to be careful with another GraphQL feature often ignored by other approaches to this problem, Directives!

What happens if you define a directive called @formatDateString and two schemes, but they have different meanings? Will this lead to unpredictable execution paths? Yes, probably. Let's fix that too.

We can rename the directive to @stripe_formatDateString And @mailchimp_formatDateString respectively. This way we can easily distinguish between the two directives.

With this all name conflicts should be resolved. Are we done yet? Not really. Unfortunately, with our solution we created many new problems!

WunderGraph: A runtime to facilitate GraphQL namespaces

By renaming all the types and fields, we actually created a lot of problems. Let's look at this query:

{
    mailchimp_customer(by: ID!) {
        id
        name
        registered @mailchimp_formatDateString(format: "ddmmYYYY")
        ... on PaidCustomer_mailchimp {
            pricePlan
        }
    }
}

What are the problems here?

Field mailchimp_customer doesn't exist in the Mailchimp schema, we have to rename it to customer.

Directive mailchimp_formatDateString also does not exist in the Mailchimp schema. We should rename it to formatDateString before being sent to the upstream. But be careful with this! Make sure this directive actually exists on the source. We automatically check this because you could accidentally use the wrong directive on the wrong field.

Finally, type definition PaidCustomer_mailchimp also does not exist in the original schema. We should rename it to PaidCustomerotherwise the source will not understand it.

Sound like a lot of work? In fact, it's already done and you can use it right now. Just enter yarn global add @wundergraph/wunderctl in your terminal and you're ready to try it out! (*deprecated, the utility was rewritten from TypeScript on GoLang)

With this we are ready for the implementation phase.

Importing API Dependencies

In the first step we need to “import” our API dependencies. We can do this using the WunderGraph SDK. Just “introspect” all the different services and combine them into an “application”.

const federated = introspect.federation({
    apiNamespace: "federation",
    upstreams: [
        {url: "http://localhost:4001/graphql"},
        {url: "http://localhost:4002/graphql"},
        {url: "http://localhost:4003/graphql"},
        {url: "http://localhost:4004/graphql"},
    ]
})

const planetscale = introspect.planetscale({
    apiNamespace: "planetscale",
    databaseURL: `mysql://${planetscaleCredentials}@fwsbiox1njhc.eu-west-3.psdb.cloud/test?sslaccept=strict`,
})

const spaceX = introspect.graphql({
    apiNamespace: "spacex",
    url: "https://api.spacex.land/graphql/",
});

const postgres = introspect.postgresql({
    apiNamespace: "postgres",
    databaseURL: "postgresql://admin:admin@localhost:54322/example?schema=public",
});

const jsonPlaceholder = introspect.openApi({
    apiNamespace: "jsp",
    source: {
        kind: "file",
        filePath: "jsonplaceholder.yaml"
    }
})

configureWunderGraphApplication({
    apis: [
        postgres,
        spaceX,
        jsonPlaceholder,
        planetscale,
        federated
    ],
});

If you look at the code you will probably notice the keyword apiNamespace repeatedly. apiNamespace ensures that each API is within its own boundary. This way, name conflicts are automatically avoided.

Once you have introspected all the dependencies, we are ready to write a query that covers all 8 services.

We want to get users from the SpaceX API, users from the JSON Placeholder API, more users from our PostgreSQL database, even more users from the Planetsacle database, and finally one user with reviews and products from the federated graph.

All this is possible thanks to our rich set of data sources.

{
    spacexUsers: spacex_users {
        id
        name
    }
    jspUsers: jsp_users {
        id
        name
        posts {
            id
            title
            comments {
                id
                body
            }
        }
    }
    postgresUsers: postgres_findManyusers {
        id
        email
    }
    planetscaleUsers: planetscale_findManyusers {
        id
        first_name
        last_name
        email
    }
    federation: federation_me {
        id
        name
        reviews {
            id
            body
            product {
                upc
                name
            }
        }
    }
}

Notice how it uses prefixed/spatial root fields. This query gives us data from all 8 services at once, it's crazy!

Now run wunderctl up, so that it all works in a couple of seconds. This runs on your local computer without calling any cloud services.

Total

In this post, we started by discussing how namespaces make it easier to write and share code. We then explored the differences between the “coding approach” and the need to deal with API integration.

We looked into Schema Stitching and Federation and learned that both approaches are good, but not enough. We looked at the One Count principle and realized that it has its drawbacks.

Finally, we introduced the concept of GraphQL API namespaces, which allows you to combine GraphQL, Federation, REST, PostgreSQL and Planetscale APIs in just a few lines of code.

If you're interested in seeing this in action, here's a video of me going through the entire process: https://youtu.be/jUaJkvPmCSQ

Our goal for WunderGraph is to become the “Package Manager for APIs.” We're not quite there yet, but eventually you'll be able to run wunderctl integrate stripe/stripethen write a Query or Mutation and the integration will be done.