creating your first distributed registry

We prepare everything you need

For experiments, you will need a computer with Ubuntu 18.04 (x64), MacOS Sierra or later versions. If you're on Windows, it's wise to look for Ubuntu installation guides. I have MacOS Sonoma at my disposal, so I pass the system requirements. If you also have a Mac, you can find out the OS version by clicking on the apple in the upper left corner and selecting “About this Mac.” And compare which is newer – your version or Sierra – using articles about MacOS on Wikipedia.

To work, you will need to install the IDE development environment. There are different options here; I have had Visual Studio Code installed for a long time. You can try it, but if you don't have an IDE yet, I recommend sticking with IntelliJ Idea. Scroll down the product page to the Community Edition, it is free and sufficient for our purposes. In the drop-down menu, select Intel or Apple Silicon, depending on the configuration of your Mac (apple in the corner – About this Mac – Processor). In the same way, select the desired installer Docker Desktop.

Both the IDE and Docker install as regular applications; It’s just important not to forget your administrator password. When Docker is successfully launched, a new Docker logo will appear in the icon set in the upper right corner.

As my further experiments showed, it will be easiest if your Mac is based on an Intel processor. Devices based on Apple architecture will require additional configuration and slightly more complex commands. I'll try to sort it out and come back to them in future posts so as not to overwhelm this one.

We deploy and launch a blockchain network

Let's create a working folder for our network. I advise you not to give it a complex name, so that its constant mention does not blur your eyes during further work. I named my working folder demo-platform.

To deploy a network you need a configuration file, docker-compose.yml. Download it to your working folder and open it using the IDE. Let’s replace the older version of the Waves Enterprise node with the newer “Confident”. To do this, replace we-network with w3-network in the entire file. And then select the text from the line in the file node-0: to line networks: (not including the last one), replace with the text below and overwrite the file:

  node-0:
    image: web3techru/confident:v1.9.0
    ports:
      - "6862:6862"
      - "6864:6864"
      - "6865:6865"
    networks:
      - w3-network
    hostname: node-0
    container_name: node-0
    env_file:
      - ./env/node-0.env
    volumes:
      - ./configs/nodes/node-0/node.conf:/node/node.conf
      - ./configs/nodes/node-0/keystore.dat:/node/keystore.dat
      - node-0-data:/node/data
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
  node-1:
    image: web3techru/confident:v1.9.0
    ports:
      - "6872:6862"
      - "6874:6864"
      - "6875:6865"
    networks:
      - w3-network
    hostname: node-1
    container_name: node-1
    env_file:
      - ./env/node-1.env
    volumes:
      - ./configs/nodes/node-1/node.conf:/node/node.conf
      - ./configs/nodes/node-1/keystore.dat:/node/keystore.dat
      - node-1-data:/node/data
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
  node-2:
    image: web3techru/confident:v1.9.0
    ports:
      - "6882:6862"
      - "6884:6864"
      - "6885:6865"
    networks:
      - w3-network
    hostname: node-2
    container_name: node-2
    env_file:
      - ./env/node-2.env
    volumes:
      - ./configs/nodes/node-2/node.conf:/node/node.conf
      - ./configs/nodes/node-2/keystore.dat:/node/keystore.dat
      - node-2-data:/node/data
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always

Initially, there will be three nodes in our network, but if projects require it, we will add more.

Now let's open a new tool for ourselves – Terminal, or command line (Programs – Utilities – Terminal). Open it and go to your working folder. To do this you will need her full address. The easiest way to get it is in Finder (Mac OS Explorer). Open your folder and see the path to it at the bottom (if you don’t see it, click View – Show path bar). Right-click on your folder in the row here and select “Copy path to…”

*You can do the same using the “Open in Terminal” option, but let's go through the basic commands, it's useful*

Return to the Terminal, enter cd and paste the path to the folder:

cd /Users/nzemlyanskiy/demo-platform

Now we are in the desired folder, we can launch the Docker container for deployment:

docker run --rm -ti -v $(pwd):/config-manager/output web3techru/config-manager:v1.9.0

After some time and many intermediate messages, we will see the line “WE network environment is ready!” in the Terminal:

Several new files and folders will appear in the working directory. Pay attention to credentials.txt, which contains information about the nodes:

Solemn moment: we can cut the ribbon of our blockchain! To do this, in the Terminal, enter:

docker-compose up -d

Nodes and services will start for some time; if successful, all lines will have positive statuses:

Now you can view network statistics. To do this, open your browser and enter in the address bar localhost or 127.0.0.1. You will be taken to the Waves Enterprise client page where you will need to create an account. Enter a custom email and password. Next, you will be asked to confirm your account by mail, but since our network is local, no confirmation is required – your user is simply added to the database on your computer. Log in with your new account and you will see information about the new network:

Three nodes, as prescribed. There is only one sender of transactions, because there is only one transaction - genesis, creation of the network. The number of blocks grows automatically while the network is running, regardless of your actions in it. — Three nodes, as prescribed. There is only one sender of transactions, because there is only one transaction – genesis, creation of the network. The number of blocks grows automatically while the network is running, regardless of your actions in it.

To prevent the network from running idle and consuming resources, you can stop it at any time with the command in the Terminal:

docker-compose down

You can start it again using the previous command.

Setting up monitoring

For future projects, we will need blockchain monitoring: this way we can track its status. We will use the open source analytics platform Grafana and the InfluxDB database.

Load Grafana using the command:

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Don't forget that in the Terminal you need to first go to our working folder. The Grafana image will be downloaded from the Internet:

Now install InfluxDB. The command below must be entered on one line:

docker run -d --name influxdb -p 8086:8086 -e INFLUXDB_DB=sandbox_influxdb -e INFLUXDB_ADMIN_USER=sandbox_influxdb_admin -e INFLUXDB_ADMIN_PASSWORD=sandbox_influxdb_pass quay.io/influxdb/influxdb:v1.6.4

For demonstration purposes, here we are using a non-latest version of InfluxDB, as indicated in the corresponding line of image. Later versions, starting with 1.8, require more complex settings, so we will focus on 1.6.4, which is still available. A successful download report should appear in the Terminal:

We return to docker-compose.yml and register the monitoring services so that they are deployed the next time the blockchain is launched. In the services section we add two new services, Grafana and InfluxDB:

grafana:
  image: grafana/grafana:latest
  hostname: grafana
  container_name: grafana
  environment:
    GF_SECURITY_ADMIN_USER: 'admin'
    GF_SECURITY_ADMIN_PASSWORD: 'pass'
  restart: always
  ports:
    - 3000:3000
  networks:
    - w3-network
  volumes:
    - grafana:/var/lib/grafana
influxdb:
  image: influxdb:1.6.4
  hostname: influxdb
  container_name: influxdb
  environment:
    - INFLUXDB_DB=influxdb
    - INFLUXDB_ADMIN_USER=admin
    - INFLUXDB_ADMIN_PASSWORD=pass
  restart: always
  ports:
    - 8086:8086
  networks:
    - w3-network
  volumes:
    - influxdb:/var/lib/influxdb

Pay attention to the fields GF_SECURITY_ADMIN_USER And GF_SECURITY_ADMIN_PASSWORD in docker-compose.yml – here we set the login and password with which we will log into Grafana. I'll stick to standard ones everywhere.

Now let's register the services in the node files. In the future, you will be able to monitor only those nodes in whose configuration monitoring will be added. In your working folder, open /configs/nodes/node-0/node.conf via IDE and add below code into it.

# Performance metrics
kamon {
  # Set to "yes", if you want to report metrics
  enable = yes
  # An interval within metrics are aggregated. After it, them will be sent to the server
  metric.tick-interval = 1 second
  # Reporter settings
  influxdb {
    hostname = "influxdb"
    port = 8086
    database = "influxdb"
    time-units = "ms"
    authentication {
      user = "admin"
      password = "pass"
    }
    environment.host = "0"
  }
}

# Non-aggregated data (information about blocks, transactions, ...)
metrics {
  enable = yes
  node-id = "0"
  influxdb {
    uri = "http://influxdb:8086"
    db = "influxdb"
    username = "admin"
    password = "pass"
    batch-actions = 100
    batch-flash-duration = 1s
  }
}

Repeat the insertion in the files /configs/nodes/node-1/node.conf And /configs/nodes/node-2/node.confchanging the values environment.host And node-id in each file on 1 And 2 respectively. Values db, username, password in node files must match the values INFLUXDB_DB, INFLUXDB_ADMIN_USER And INFLUXDB_ADMIN_PASSWORD in docker-compose.yml.

Return to docker-compose.yml. Add InfluxDB three times, in blocks node-0, node-1 And node-2 after the line image…:

depends_on:
  - influxdb

And finally, list the services at the very end, in the volumes section, adding:

grafana:
influxdb:

If your blockchain is already running, restart it using the command

docker-compose up -d

It is important that each line in docker-compose.yml is properly indented. The first level list contains services (grafana, influxdb, nginx-proxy…). Each of them has a nested list of settings (image, hostname, container_name…). If you move the service to the settings level, Docker will think that this is an invalid setting and will throw an error:

I’ll share another error that I encountered at this stage:

It seems that as a result of some unsuccessful attempt, the containers started partially, and subsequently a conflict arose when trying again. The solution here is simple, go to Docker Desktop through the icon in the dock or in the top panel (Go to the Dashboard):

This is a list of running containers. Do you recognize the names from docker-compose? Your list may have a different order, the main thing is to remove all containers, for which there is a trash can icon on each line on the right. After this, restart the network:

Setting up the Grafana web client

Let's go to the browser to configure Grafana. She is located at http://127.0.0.1:3000/login. To log in, use the login and password you set earlier (in my case it’s just admin/pass).

Open the sandwich menu in the upper left corner, click Connections – Add new connection. Enter InfluxDB in the search bar, select this database and enter the following data on the settings page:

Query language: InfluxQL (this language is compatible with our version of InfluxDB)
URL: http://influxdb:8086 (we specified this port in docker-compose)
Database: influxdb (INFLUXDB_DB value from docker-compose)
User: admin (INFLUXDB_ADMIN_USER value)
Password: pass (INFLUXDB_ADMIN_PASSWORD value)

Click Save & Test. In response, you should receive a green die. Due to delays you may have a different number of metrics, it is important that Grafana confirms a successful connection:

Now, based on the connected data, we will create a dashboard. Open the menu in the upper left corner, then Dashboards – New – New Dashboard – Import Dashboard. In the Import via dashboard JSON model field, copy code from documentation. Click Load, in the next step we assign a name to the dashboard and determine its place in the dashboard hierarchy (I left the Dashboards root folder). Click Import.

Now monitoring of our blockchain is available in the menu – Dashboards. In the panel you will see many graphs that are empty because they track transactions and other events that we are not using at this stage. For now, the first graph showing the height of the blockchain is important for us:

The screenshot above demonstrates an ideal monitoring picture, when all three nodes synchronously gain the same height. In fact, differences are possible: Grafana may not immediately pull up all the nodes, and their height may vary if you restarted and rebuilt a lot of things in the process.

Additionally, the configuration of your computer plays a big role here. For example, I was unable to achieve perfection on a Macbook Pro 2018 with Intel i7 and 16 GB of RAM. The screenshot above was sent by my devops colleague (thanks, Alik!) with the build on a Mac M1. It’s okay, monitoring is a derivative of the network’s operation, and it cannot influence it. In future projects, you will restart the blockchain more confidently and, perhaps, get closer to the ideal grafana 🙂 At the acquaintance stage, it is important that the network is functioning and we see it.

By the way, if you don’t see this, then perhaps this is only due to the Grafana configuration. Specify the desired time interval at the top right and turn on auto-update using the circular arrow button.

Let's sum it up

Congratulations: we have created our own blockchain and even deployed monitoring! In the following posts, I will try to figure out how to repeat this success on Macs with Apple Silicon, perhaps I will move to the newer InfluxDB, and we will move on to demo projects.

If you have any questions, ask them in the comments. If I can’t answer it myself, I’ll try to involve our experienced developers and devops. I will be very glad to receive comments on how anything can be optimized in the instructions. Until new challenges!