Expanding and moving the ZooKeeper ensemble

I welcome everyone! Recently, our DBA team was faced with the task of moving the ZooKeeper ensemble to other servers. The problem turned out to be that there is no manual available on the Internet for ZooKeeper version 3.6.2 and higher. Found management for version 3.4.5, but in this way we did not dare to expand and move our ensemble.

The question may arise: Why can’t you just move the files to another server and run the ensemble there? Our ensemble stores data for the merge and replication of cluster data Clickhouse

Work plan

We have an ensemble of 3 instances located on 3 separate servers and have been provided with 3 other servers to move. In order to maintain a quorum, we found 1 additional temporary server. First of all, we will expand the cluster to 7 instances:
3 (id: 1, 2, 3) +1 (id: 4) + 3 (id: 5, 6, 7)

After expansion and synchronization, we remove unnecessary instances from the ensemble. Only id will remain: 5, 6, 7.

Expansion of an existing ensemble

The official documentation describes method extensions, but we want to show it clearly.
Our ensemble runs on CentOS 7 and consists of 3 instances:

server.1=zk-1:2888:3888
server.2=zk-2:2888:3888
server.3=zk-3:2888:3888	

If you do not have in the configuration file no

reconfigEnabled=true

then you need to restart your ensemble with this line. Also to file $ ZK_HOME / bin / zkEnv.sh you can add:

Dzookeeper.skipACL=yes
export SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Dzookeeper.skipACL=yes"

these lines will be needed when performing dynamic ensemble reconfiguration.
After restarting the ensemble, we can start expanding. To do this, we will prepare everything you need:

  1. Download the archive with apache-zookeeper (in our case, apache-zookeeper-3.6.2-bin.tar.gz);

  2. Unpack and create an OS user for ZooKeeper;

  3. Create a directory for ZooKeeper files and a myid file that will be the instance identifier. We already have 3 instances server.1=zk-1:2888:3888 server.2=zk-2:2888:3888 server.3=zk-3:2888:3888… We plan to add 4 more instances to expand. Therefore, myid will contain the numbers 4, 5, 6, 7;

  4. Open ports in firewalld;

  5. Create a service file.

After preparation, we need $ ZK_HOME / bin / zkCli.sh

reconfig -add server.4=zk-4:2888:3888:participant;2181

The success of the execution can be checked in $ ZK_HOME / conf. The file zoo.conf.dynamic.100000000 will appear and the corresponding link to this file in zoo.conf:

dynamicConfigFile=/home/zookeeper/apache-zookeeper-3.6.2-bin/conf/zoo.conf.dynamic.100000000

The number after zoo.conf.dynamic. will be different. Next, we launch the ZooKeeper instance on the zk-4 server. This instance will receive the same configuration files. You can follow the progress of synchronization in the logs. We perform the same operation for servers zk-5, zk-6, zk-7.

Conclusion from the ensemble

To output zk-1, zk-2, zk-3, zk-4 from the ensemble, we use the same $ ZK_HOME / bin / zkCli.sh

reconfig -remove 1
reconfig -remove 2
reconfig -remove 3
reconfig -remove 4

After execution, you will also see how the zoo.conf.dynamic.XXXXXXX file will change. Now you can turn off the ZooKeeper service on the zk-1, zk-2, zk-3, zk-4 servers. Do not forget to remove from autorun.

Total

In total, by expanding with new instances and withdrawing old ones, we moved the ensemble from the zk-1, zk-2, zk-3 servers to the zk-5, zk-6, zk-7 servers.
Additionally, I advise you to add to zoo.conf and open port 7000

metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider

Thus, you will be able to collect metrics through Prometheus.

Similar Posts

Leave a Reply