Almost everything about Carbonio Mesh
Carbonio Mesh is an infrastructure module of the Carbonio mail server that allows for connectivity, resiliency and monitoring of various Carbonio services and their integration with Carbonio LDAP. In this article we will explain what the Carbonio Mesh system consists of and how to ensure effective interaction with it.
This article is suitable for both users of the commercial version of Carbonio and users of Carbonio Community Edition.
Carbonio Mesh is a mandatory Carbonio element that is installed and configured on each node during the mail server installation process and provides a process for automatically discovering and configuring the connectivity of various services with the Carbonio mail server.
Carbonio Mesh can be installed in two versions – server and agent. The difference between them is that the agent cannot work without active servers, and the active server can continue to work if communication with the Carbonio Mesh agents is lost.
Communication between Carbonio Mesh servers and agents occurs based on credentials, a file with which is stored on each Carbonio Mesh server in the path /etc/zextras/service-discover/cluster-credentials.tar.gpg.
If this file falls into the hands of cybercriminals, the entire Carbonio mail server will be at risk. That is why it is recommended to install Carbonio Mesh servers on nodes that are inaccessible from public networks. For example, to an LDAP server or a node with PostgreSQL databases.
In addition to credentials, Carbonio Mesh uses its own certificate issuance system, as well as a secret key, which is used to auto-configure Carbonio Mesh and services on added infrastructure nodes. The secret key is set when the first Carbonio Mesh server is installed and then stored in clear text in the file /var/lib/service-discover/password. The file can only be viewed by the root user.
If your credentials are compromised, you can create new ones. To do this, get a list of all service-discover servers using the command carbonio prov getAllServers service-discover and enter the command wget http://127.0.0.1:8500/v1/status/leader -qO –to make sure you are on the leading server. Even if you have installed several Carbonio Mesh servers, only one will be the leader. If the IP address in the command output matches the address of the host you are on, everything is fine. If this is not the case, go to the Carbonio Mesh server whose address is indicated.
While on the leading server, enter the command consul acl bootstrap. Its output will contain the value needed to reset the token. For example, Failed ACL bootstrapping: Unexpected response code: 403 (Permission denied: ACL bootstrap no longer allowed (reset index: 908))
Knowing this value, write it to the appropriate file sudo -u service-discover bash -c “echo 908 > /var/lib/service-discover/data/acl-bootstrap-reset”. Let's stop Carbonio Mesh services systemctl stop service-discover and delete files with settings and previously issued certificates.
rm /etc/zextras/service-discover/config.json
rm /etc/zextras/service-discover/main.json
rm /var/lib/service-discover/*.pem
After this, you need to set a new secret key using the command
service-discover setup LEADER_IP –first-instance –password=S3cr3T
By replacing LEADER_IP with the IP address of the leading server, and S3cr3T with your complex password.
After this, the leading server will be restored to work with the new credentials, and the remaining Carbonio Mesh nodes can be reconfigured to work with the updated server.
The Carbonio Mesh graphical interface can be used to monitor services. For security reasons, the Carbonio Mesh web interface accepts connections on port 8500 only from localhost, so to access it you need to forward an ssh tunnel using a command like
ssh -N -f -L 8400:localhost:8500 root@IP
Replace the IP in the command with the one you use. You may also need to enable SSH password access for the root user.
After forwarding the SSH tunnel, the Carbonio Mesh web interface can be opened in a browser at http://localhost:8400/
It can be accessed without providing any credentials. The main page displays a list of available services and their status.
In addition to the main list of services, you can view various details about each of them.
The Carbonio Mesh web interface allows you to visually view the cluster structure and identify problem areas.
You can also manage Carbonio Mesh from the command line. This is done using HashiCorp Consul commands, which is the basis of Carbonio Mesh.
For example, to obtain various lists of the constituent parts of Carbonio Mesh, a command like consul catalog is used. For example, to view the services running on the main node, you need to get a list of nodes, and then get a list of services, indicating the name of the desired node. This is done using the commands
consul catalog nodes
consul catalog services -node b9bb7ab4
To get a list of servers and agents that are part of Carbonio Mesh, use the command consul members.
To remove nodes from the list, for example in case of IP address changeuse the command consul force-leave server-mail-carbonio-local
To remove individual services, use a command like consul services deregister -id=carbonio-docs-connector.
To reboot Carbonio Mesh use the command consul reloadand to view logs, journalctl -u service-discover.
Thus, Carbonio Mesh is a flexible solution that allows you to automatically combine different services on different servers into a single cluster so that users can conveniently use them directly from the Carbonio web client
