adding custom plugins using docker-compose

Gravitee.io

Gravitee.io – open source gateway product with API showcase. This article is intended for those who are already familiar with the system. General information about the product is well described in the article: What are API Management systems.

Most of the cases for working with queries can be done using existing plugins, but you can write your own if you wish. If you get carried away in this matter, then you come to the conclusion that it is easier to write your own plugin than to deal with existing ones. Sometimes using built-in plugins is quite a tricky art, but if you stumbled across this article, you already know that.

Also, writing your own plugins adds a lot of possibilities when working with Gravitee

About what?

In this article, we will not consider how to write your own plugins, we will rely on the fact that you are learning how to do it and have achieved some success, and now you want to connect, test and use them. As a bonus, we will learn how to log in Graylog.

Let’s get started

I will perform my manipulations on the latest version of Gravitee at the time of publication (3.20.10), but everything described below is also true for previous versions.

First, let’s take docker-compose.yml from the official source and start making changes.

Install the required version of Gravitee

Let’s forward the folder for custom plugins to the volumes section for management_api:
– ./apim-management-api/plugins:/opt/graviteeio-management-api/plugins-ext
Here we set the path of the folder with plugins within our system, as well as within the container

In the environment section for management_api, add the paths to our plugins:
– gravitee_plugins_path_0=/opt/graviteeio-management-api/plugins
– gravitee_plugins_path_1=/opt/graviteeio-management-api/plugins-ext

Similarly, we will forward the folder for custom plugins to the volumes section for gateway:
– ./apim-gateway/plugins:/opt/graviteeio-gateway/plugins-ext

Similarly, in the environment section for gateway, add the paths to our plugins:
– gravitee_plugins_path_0=/opt/graviteeio-gateway/plugins
– gravitee_plugins_path_1=/opt/graviteeio-gateway/plugins-ext

The finished docker-compose.yml file is shown below:

docker-compose.yml

version: '3.5'

networks:
  frontend:
    name: frontend
  storage:
    name: storage

volumes:
  data-elasticsearch:
  data-mongo:

services:
  mongodb:
    image: mongo:3.6
    restart: always
    ports:
      - "27017:27017"
    volumes:
      - data-mongo:/data/db
      - ./logs/apim-mongodb:/var/log/mongodb
    networks:
      - storage

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: gio_apim_elasticsearch
    restart: always
    volumes:
      - data-elasticsearch:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=0.0.0.0
      - xpack.security.enabled=false
      - xpack.monitoring.enabled=false
      - cluster.name=elasticsearch
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile: 65536
    networks:
      - storage

  gateway:
    image: graviteeio/apim-gateway:3.20.10
    container_name: gio_apim_gateway
    restart: always
    ports:
      - "8082:8082"
    depends_on:
      - mongodb
      - elasticsearch
    volumes:
      - ./logs/apim-gateway:/opt/graviteeio-gateway/logs
      - ./apim-gateway/plugins:/opt/graviteeio-gateway/plugins-ext
    environment:
      - gravitee_management_mongodb_uri=mongodb://mongodb:27017/gravitee?serverSelectionTimeoutMS=5000&connectTimeoutMS=5000&socketTimeoutMS=5000
      - gravitee_ratelimit_mongodb_uri=mongodb://mongodb:27017/gravitee?serverSelectionTimeoutMS=5000&connectTimeoutMS=5000&socketTimeoutMS=5000
      - gravitee_reporters_elasticsearch_endpoints_0=http://elasticsearch:9200
      - gravitee_plugins_path_0=/opt/graviteeio-gateway/plugins
      - gravitee_plugins_path_1=/opt/graviteeio-gateway/plugins-ext
    networks:
      - storage
      - frontend

  management_api:
    image: graviteeio/apim-management-api:3.20.10
    container_name: gio_apim_management_api
    restart: always
    ports:
      - "8083:8083"
    links:
      - mongodb
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    volumes:
      - ./logs/apim-management-api:/opt/graviteeio-management-api/logs
      - ./apim-management-api/plugins:/opt/graviteeio-management-api/plugins-ext
    environment:
      - gravitee_management_mongodb_uri=mongodb://mongodb:27017/gravitee?serverSelectionTimeoutMS=5000&connectTimeoutMS=5000&socketTimeoutMS=5000
      - gravitee_analytics_elasticsearch_endpoints_0=http://elasticsearch:9200
      - gravitee_plugins_path_0=/opt/graviteeio-management-api/plugins
      - gravitee_plugins_path_1=/opt/graviteeio-management-api/plugins-ext
    networks:
      - storage
      - frontend

  management_ui:
    image: graviteeio/apim-management-ui:3.20.10
    container_name: gio_apim_management_ui
    restart: always
    ports:
      - "8084:8080"
    depends_on:
      - management_api
    environment:
      - MGMT_API_URL=http://localhost:8083/management/organizations/DEFAULT/environments/DEFAULT/
    volumes:
      - ./logs/apim-management-ui:/var/log/nginx
    networks:
      - frontend

  portal_ui:
    image: graviteeio/apim-portal-ui:3.20.10
    container_name: gio_apim_portal_ui
    restart: always
    ports:
      - "8085:8080"
    depends_on:
      - management_api
    environment:
      - PORTAL_API_URL=http://localhost:8083/portal/environments/DEFAULT
    volumes:
      - ./logs/apim-portal-ui:/var/log/nginx
    networks:
      - frontend

The next step is to bring Gravitee up using docker-compose up -d, or manually create plugin paths in the directory with our docker-compose.yml file.
Path for gateway: /apim-gateway/plugins
Path for management: /apim-management-api/plugins

Adding a Plugin

After creating the necessary paths, we need to build our plugin. Go to the target folder and take the zip archive obtained during the build. The plugin based on which I worked will be available on GitHub

Now we place our plugin in both created directories and start/restart Gravitee. An important clarification: after each update / addition of the plugin, you will need to restart Gravitee or separately its gateway and management-api modules

After launch, we need to create an API, go to the Design tab and find our plugin in the right list. How it looks in my case:

The plugin category depends on which category you put it in with plugin.properties

Your plugin is now ready to use! You can start configuring. Don’t forget to do “deploy your api” after every plugin configuration change.

Bonus

Logging in Graylog cannot be configured separately for a specific plugin, but it can be configured for a separate component such as management, gateway and others.

Each component has a config directory that contains the logback.xml file, which we will be looking at, as well as gravitee.yml (an important config file) and some other files depending on the module that we will not be looking at.

Initially, the logback.xml file looks like this:

logback.xml

<!--
  ~ Copyright (c) 2015-2016, The Gravitee team (http://www.gravitee.io)
  ~
  ~  Licensed under the Apache License, Version 2.0 (the "License");
  ~  you may not use this file except in compliance with the License.
  ~  You may obtain a copy of the License at
  ~
  ~  http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~  Unless required by applicable law or agreed to in writing, software
  ~  distributed under the License is distributed on an "AS IS" BASIS,
  ~  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~  See the License for the specific language governing permissions and
  ~  limitations under the License.
  -->

<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] [%X{api}] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${gravitee.home}/logs/gravitee.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- daily rollover -->
            <fileNamePattern>${gravitee.home}/logs/gravitee_%d{yyyy-MM-dd}.log</fileNamePattern>

            <!-- keep 30 days' worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>

        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] [%X{api}] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="async-file" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="FILE" />
    </appender>

    <appender name="async-console" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="STDOUT" />
    </appender>

    <logger name="io.gravitee" level="INFO" />
    <logger name="com.graviteesource" level="INFO" />
    <logger name="org.reflections" level="WARN" />
    <logger name="org.springframework" level="WARN" />
    <logger name="org.eclipse.jetty" level="WARN" />

    <!-- Strictly speaking, the level attribute is not necessary since -->
    <!-- the level of the root level is set to DEBUG by default.       -->
    <root level="WARN">
        <appender-ref ref="async-console" />
        <appender-ref ref="async-file" />
    </root>

</configuration>

We need to add configurations depending on how we will be logging to Graylog. For example, in the case of UDP, we will add:

    <appender name="GELF" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:host</host>
        <port>port</port>
        <facility>НАЗВАНИЕ КОМПОНЕНТА</facility>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <includeFullMdc>true</includeFullMdc>
    </appender>


    <logger name="io.gravitee.rest.api.service.impl.upgrade" level="INFO">
        <appender-ref ref="FILE-UPGRADERS"/>
    </logger>
    <logger name="io.gravitee" level="INFO"/>
    <logger name="org.eclipse.jetty" level="INFO"/>
    <logger name="org.reflections" level="WARN" />
    <logger name="org.springframework" level="WARN" />
    <logger name="org.eclipse.jetty" level="WARN" />

    <!-- Strictly speaking, the level attribute is not necessary since -->
    <!-- the level of the root level is set to DEBUG by default.       -->
    <root level="WARN">
        <appender-ref ref="GELF"/>
        <appender-ref ref="async-console" />
        <appender-ref ref="async-file" />
    </root>

Added logback.xml

<!--
  ~ Copyright (c) 2015-2016, The Gravitee team (http://www.gravitee.io)
  ~
  ~  Licensed under the Apache License, Version 2.0 (the "License");
  ~  you may not use this file except in compliance with the License.
  ~  You may obtain a copy of the License at
  ~
  ~  http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~  Unless required by applicable law or agreed to in writing, software
  ~  distributed under the License is distributed on an "AS IS" BASIS,
  ~  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~  See the License for the specific language governing permissions and
  ~  limitations under the License.
  -->

<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] [%X{api}] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${gravitee.home}/logs/gravitee.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- daily rollover -->
            <fileNamePattern>${gravitee.home}/logs/gravitee_%d{yyyy-MM-dd}.log</fileNamePattern>

            <!-- keep 30 days' worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>

        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] [%X{api}] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="async-file" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="FILE" />
    </appender>

    <appender name="async-console" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="STDOUT" />
    </appender>

    <appender name="GELF" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:host</host>
        <port>port</port>
        <facility>НАЗВАНИЕ КОМПОНЕНТА</facility>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <includeFullMdc>true</includeFullMdc>
    </appender>


    <logger name="io.gravitee.rest.api.service.impl.upgrade" level="INFO">
        <appender-ref ref="FILE-UPGRADERS"/>
    </logger>
    <logger name="io.gravitee" level="INFO"/>
    <logger name="org.eclipse.jetty" level="INFO"/>
    <logger name="org.reflections" level="WARN" />
    <logger name="org.springframework" level="WARN" />
    <logger name="org.eclipse.jetty" level="WARN" />

    <!-- Strictly speaking, the level attribute is not necessary since -->
    <!-- the level of the root level is set to DEBUG by default.       -->
    <root level="WARN">
        <appender-ref ref="GELF"/>
        <appender-ref ref="async-console" />
        <appender-ref ref="async-file" />
    </root>

</configuration>

Now it remains only to supplement docker-compose.yml by dropping the config folder inside the container. For example for management-api it would be like this:

- .\apim-management-api\config:/opt/apim-management-api/config

This configuration can be applied to all Gravitee modules and everyone will write to Graylog together.

The next step is to “enrich” our Gravitee with the biz.paluch.logging.logstash-gelf library. To do this, we will turn to maven-central, where we will get the jar of the library and place it along the path /opt/название компонента/lib (within the container). To do this, we will again “forward” the folder outside, for a single or all components. For example, for the gateway component:

- .\apim-gateway\lib:/opt/graviteeio-gateway/lib

It remains only to configure Graylog, which I will leave to the reader’s conscience

Conclusion

In this article, we looked at how to add custom plugins to Gravitee.io using docker-compose.yml. We also taught our Gravitee how to write in Graylog as a bonus. The plugin source code and docker-compose.yml (pre-bonus version) are available at GitHub