One of our clients – a federal-scale retailer – took care of backing up data in the Oracle DBMS. Oracle Zero Data Loss Recovery Appliance (ZDLRA) is the default for this. But the complex is like a cruise icebreaker. In addition, ZDLRA would not give the client control over all backup processes through a single console. These considerations forced us to look for alternatives. One of them is the Veritas NetBackup Appliance 5240, a mid-range SRC with good performance under standard conditions. The Copilot technology in Veritas’ arsenal, designed specifically for working with Oracle DBMS, also added optimism.
Before testing the Veritas NetBackup Appliance 5240 on a live infrastructure, the customer asked to test it. We assembled the stand and tested the solution in combat conditions. The conclusions turned out to be interesting.
Pros of Veritas NBU
We first looked at what unique technologies can speed up the backup and recovery process. Since we were talking about backing up an Oracle database and using 10 GbE (no Fiber Channel) as the network connection, the following Veritas tools were most useful:
- Media Server Deduplication Pool (MSDP) – data deduplication “on the fly”, which optimizes replication of backups between devices and creates synthetic full backups during incremental ones;
- NetBackup Optimized Duplication eliminates the redundancy of the transmitted data by transmitting only unique blocks that are absent on the receiving device;
- NetBackup Copilot Reduces the time it takes to create Oracle DBMS backups with snapshots of the NetBackup appliance file system and integration with Oracle RMAN Backup Manager.
Most of the hope in the context of Oracle was NetBackup Copilot. In testing, we focused on checking its performance against regular incremental database copies.
Are you ready for testing? Yes but no
We deployed a test bed that included NetBackup Master Server, NetBackup Media Server, and Oracle Linux Server 6.7. The NetBackup Appliance (acting as the NetBackup Media Server) was connected to the database via two 10 GbE ports, and the NetBackup Master Server was deployed on a virtual machine in a VMware vSphere 6.0 virtualization environment.
A physical server with installed OS Oracle Linux Enterprise 6.7 and DBMS Oracle 19 was used as a source of RK. To simulate the operation of the system in conditions close to the customer’s requirements, we set the volume of the Oracle test base in the size of 1 TB in Bigfile format. The database was under load, and the volume of changes within 12 hours was 50-60% of the original volume of the database.
So let’s go! We ran the backup, but the performance level was surprisingly low – 2.3-2.8 TB / h. According to the results – hello from the 90s! The documents on the work of Veritas NBU with the Oracle DBMS did not contain ready-made solutions for this situation. But the very fact of Copilot availability and the good performance of the solution on standard tasks, such as backing up file systems, suggested that we were missing some points. Then, together with colleagues from Veritas, we began to search for tweaks to NetBackup that would improve performance.
We checked several dozen settings and found the optimal values for them. Among the parameters that influenced the performance of the test bench were:
- Jumbo Frame values (sizes of Ethernet frames in which data can be transferred);
- hash transfer policy (xmit_hash_policy), which directly affects the speed and efficiency of backups;
- the size of the buffers (Number Disk, Size Disk) of the Veritas Appliance needed to be changed to back up a frequently changing database
Should I use Copilot?
We had high hopes for NetBackup Copilot – after all, this technology was originally designed to work with a database and uses Oracle incremental merge to move to a forever incremental backup scheme. When operating in Copilot mode, the system interacts with the Oracle RMAN DBMS backup manager to run the DBMS backup commands.
If you break down the backup process using NetBackup Copilot into stages, it looks like this:
- A backup storage device is configured on the NetBackup Appliance that is accessible to the Oracle Database Server over the NFS protocol;
- after that, the backup policy is configured in the administration console of the NetBackup SRK;
- the backup process itself begins with creating a base copy (level-0), after which only incremental backups (level-1) are performed;
- Changes since the baseline level-0 backup was created are applied to the image that is updated at the time of the last level-1 backup.
- NetBackup creates snapshots of the device’s NFS backup image (using the functionality of the device-integrated InfoScale software);
- each backup and snapshot is added to the Oracle RMAN backup manager catalog and NetBackup service database.
This solution has many advantages. For example, snapshots of the NFS storage file system of a NetBackup Appliance can be automatically replicated (moved) to the most efficient storage tier: disk, deduplication pool, tape, cloud storage, or replicated to a NetBackup appliance at a backup site. This is done through lifecycle management policies (SLP).
In addition, DBMS administrators can use Oracle backup and restore utilities. Incremental backups allow you to work with a large number of restore points, and all copies are in file storage that does not need to be managed.
And if the speed?
How fast does it all work? After optimizing and manually tweaking individual parameters, we got a pretty decent backup speed.
The table summarizes the results of creating a full backup with deduplication enabled and disabled on the client, and Redo logs trimming enabled and disabled, in conditions when the DBMS is under load and without load.
|Type||Job Schedule||DB Load||Client deduplication||Redo logs||Elapsed time||Speed TB / h|
The NBU backup system showed good write speed of backups. The obvious bottleneck in our test was the Veritas Appliance disk subsystem in the 5240 model (the number of disks in the RAID group and the interface speed). The tests used a minimal configuration with one disk enclosure.
Making incremental copies
To evaluate the performance in the incremental backup mode, we performed backups twice a day at 10:00 and 22:00. The DBMS was under load and deduplication was enabled on the client.
|Type||Job Schedule||DB Load||Client deduplication||Elapsed time||Speed TB / h|
The incremental backup times were much shorter, but the speed of the backup sessions was also slower.
Turn on Copilot mode
The situation looks different in Copilot mode. In our test, a backup was created every 12 hours, and the backup time was recorded from the moment the Oracle snapshot was created until the end of the moment the backup was written to the storage pool on the NBU device.
|Type||DB Load||Elapsed time||Megabytes||Speed TB / h|
|Backup||Yes||0:32:14||1 126 525||2.5|
|Backup||Yes||0:33:34||1 152 365||2.7|
|Backup||Yes||0:31:23||1 123 620||2.6|
|Backup||Yes||0:44:04||1 681 999||2.9|
The results of this test were average. However, it should be borne in mind that the synthesis of the backup with subsequent writing to the storage pool took place in NFS Share. Additional NFS Share read and write speed limits may be partly responsible for the poor performance. In addition, there is Optimized Share technology for the “older” NetBackup Appliance models, so the speed of operation in this mode should be higher. We used the Veritas Appliance in a minimal configuration with one shelf, while the vendor recommends a minimum of two shelves for Copilot mode.
Thus, the main benefit of using Copilot is restoring the last full backup without having to roll forward incremental backups. Using the Instant Restore function for quick access to the DBMS while still in the recovery process is also a big plus.
Not more than 25% and within 50 TB
Let’s go back to the customer case. Testing on a synthetic database turned out to be useful, as it helped the client see all the pros and cons of the initially attractive solution. After playing with the parameters, we came to the conclusion that using Veritas NetBackup is advisable for a DBMS up to 50 TB in size, as well as with daily changes in the database of no more than 25%. With retail databases changing by 50% every day, Veritas NetBackup was not a viable solution.
The side effect of our testing has proven to be valuable. We have found the optimal modes for Veritas NBU to work together with Oracle DBMS. By adjusting the parameters and choosing the mode (classic copy or Copilot), you can create a worthy and more affordable alternative for backing up and restoring Oracle DBMS with a relatively small number of daily changes in the database in the tens of TB. For those who already use the Veritas SRK, this is the best solution. This is the use of a more affordable SRK and management of all backups through one console.
Author: Artem Khmelenko, Engineer of Data Storage Systems, Jet Infosystems