Segment replication in OpenSearch
Many of our colleagues are increasingly looking towards OpenSearch, which is gradually acquiring more and more new features. In the telegram channel we are already published a post describing updates in version 2.7, among which there is replication of segments (there is also a search in snapshots, but about it some other time). Shard replication is an alternative to document replication. When replicating documents, all replica nodes perform the same indexing operation as the master node. With shard replication, only the master node performs the indexing operation, creating shard files that are then copied to each replica node. With this replication scheme, the indexing load falls only on the main node, freeing up resources on the replicas to be used for other operations. In this post, we will explain the concept of shard replication, advantages and disadvantages compared to document replication. Wellcome that tackle.
When creating an index in OpenSearch, the number of main shards (number_of_shards is 1 by default) and the number of replicas (number_of_replicas is 1 by default) are set. Each replica is a complete copy of the main shard. If you have 5 main shards and 1 replica for each of them, then there are 10 shards in the cluster in the index. Each shard is an instance of a Lucene index, a Java library for reading and writing index structures. Lucene is a file (segment) based search engine that can be added to but not deleted. A segment is a portion of a Lucene index as a file on disk. Each document that is indexed is divided into fields. Lucene keeps this data in RAM until it is written to disk in those same segments. Well, replicas provide redundant copies of data for fault tolerance and speed up searches in the index.
Replication of OpenSeacrh documents
For versions 2.7 and earlier, document replication is the default replication mode. In this mode, all write operations that affect the index (for example, adding, updating, or deleting documents) are first directed to the node containing the main index shard. The main shard is responsible for checking the indexing operation. After the indexing operation successfully completes, the data is sent in parallel to each node in the replica group. Each node with a replica performs the same operation, duplicating the processing performed on the node with the main shard. When the operation completes on the replica (successfully or with an error), the response is sent to the node with the main shard. After all replicas have responded, the node with the main shard replies to the coordinating node, which then replies to the client with a success or failure status for the replication.
The advantage of document replication is that documents become searchable on replicas more quickly, as they are sent to replicas immediately after they arrive on the main shard. The cluster reaches a consistent state between the main shards and replicas faster. At the same time, document replication consumes more processor power, because indexing operations are duplicated on nodes with a main shard and replicas for each document. Below is just that process.
Replication of OpenSeacrh segments
When replicating shards, documents are indexed only on the node containing the main shard. The generated shard files are then copied to all replicas in the replication group. Shard replication reduces the CPU load when indexing, updating or deleting documents due to the fact that indexing is performed on only one node.
Document replication vs. segment replication
Shard replication trades CPU usage for time and network. The main shard sends large blocks of data to its replicas less frequently. As the number of replicas increases, the main shard becomes the bottleneck, doing all the indexing work and replicating all the shards. During testing, a stable improvement was observed with the number of replicas equal to one. As the number of replicas increases, the improvement decreases linearly. Performance improvement in a cluster depends on the workload and configuration.
Shard replication is best suited for the following configurations:
- The cluster has a small number of replicas (1-2 replicas).
- There is a lot of data coming into the cluster and few lookups being performed.
- The application is not sensitive to replication latency.
- The network bandwidth between the nodes is sufficient for a large amount of data transfer.
Document replication is more suitable in the following cases:
- The cluster has a large number of replicas (more than 3) and low replication latency is important to you.
- The cluster does not tolerate replication latency. Important for intensive searches where data consistency across all replicas is critical.
- Cluster nodes have insufficient network bandwidth to transfer data quickly.
The performance of shard replication is on average 40% higher than that of document replication with the same cluster configuration. With shard replication, you can get the same indexing performance with 9 nodes in a cluster as with 15 nodes with document replication. Yes, and enabling shard replication for an existing index requires reindexing. For more information about performance tests, see OpenSearch blog.
