Using operations hooks to backup files on macOS on the fly

Hello, Habr! My name is Denis Kopyrin, and today I want to talk about how we solved the problem of backup on demand on macOS. In fact, an interesting task that I encountered at the institute eventually grew into a large file system project in macOS and became part of the Acronis Active Protection system. All the details are under the cut.

I will not start from afar, I can only say that it all started with a project at the Moscow Institute of Physics and Technology, which I developed with my supervisor at the Acronis base department. We were faced with the task of organizing remote file storage, or rather, maintaining the current status of their backups.

To ensure data safety, we use the macOS kernel extension, which collects information about events in the system. KPI for developers has a KAUTH API, which allows you to receive notifications about opening and closing a file – that’s all. If you use KAUTH, you must completely save the file when opening it for writing, because the events of writing to the file are not available to developers. Such information was not enough for our tasks. Indeed, in order to permanently supplement a backup copy of data, you need to understand exactly where the user (or malware 🙂 wrote the new data to the file.

But which of the developers were scared by OS restrictions? If the kernel API does not allow you to obtain information about write operations, then you need to come up with your own way of intercepting through other kernel tools.

At first, we did not want to patch the core and its structures. Instead, they tried to create a whole virtual volume that would allow us to intercept all read and write requests passing through it. But it turned out one unpleasant feature of macOS: the operating system believes that it does not have 1, but 2 USB flash drives, two disks, and so on. And from the fact that the second volume changes when working with the first, macOS begins to work incorrectly with drives. There were so many problems with this method that I had to abandon it.

Search for another solution

Despite the limitations of KAUTH, this KPI allows you to get notified about the use of a file for recording before all operations. Developers are given access to the BSD file abstraction in the kernel – vnode. Oddly enough, it turned out that patching vnode is easier than using volume filtering. The vnode structure has a table of functions that provide work with real files. Therefore, we had the idea to replace this table.

The idea was immediately regarded as a good idea, but for its implementation it was necessary to find the table itself in the vnode structure, since Apple does not document its location anywhere. To do this, it was necessary to study the machine code of the kernel, and also to figure out whether it is possible to write to this address so that the system does not die after that.

If the table is found, we simply copy it into memory, replace the pointer and paste the link to the new table into the existing vnode. Thanks to this, all operations with files will go through our driver, and we will be able to register all user requests, including read and write. Therefore, the search for the treasured table has become our main goal.

Given that Apple does not really want this, to solve the problem you need to try to “guess” the location of the table using heuristics for the relative location of the fields, or take an already known function, disassemble it and look for an offset from this information.

How to look for an offset: an easy way

The simplest way to find the offset of a table in vnode is a heuristic, which is based on the location of the fields in the structure (link to github)

struct vnode
{
  ...
  int (**v_op)(void *); /* vnode operations vector */
  mount_t v_mount; /* ptr to vfs we are in */
  ...
}

We will use the assumption that the v_op field we need is exactly 8 bytes removed from v_mount. The value of the latter can be obtained using public KPI (link to github):


mount_t vnode_mount(vnode_t vp);

Knowing the value of v_mount, we begin to look for a “needle in the haystack” – we will perceive the value of the pointer to vnode ‘vp’ as uintptr_t *, the value of vnode_mount (vp) as uintptr_t. This is followed by iterations to the “reasonable” value of i, until the condition ‘haystack[i]== needle ’. And if the assumption about the location of the fields is correct, the offset v_op is i-1.

void* getVOPPtr(vnode_t vp)
{
  auto haystack = (uintptr_t*) vp;
  auto needle = (uintptr_t) vnode_mount(vp);
  for (int i = 0; i < ATTEMPTCOUNT; i++)
  {
    if (haystack[i] == needle)
    {
      return haystack + (i - 1);
    }
  }
  return nullptr;
}

How to look for an offset: disassembling

Despite its simplicity, the first method has a significant drawback. If Apple changes the order of the fields in the vnode structure, the simple method will break. A more universal, but less trivial method is to dynamically disassemble the kernel.

For example, consider the disassembled kernel function VNOP_CREATE (link to github) on macOS 10.14.6. Instructions that are interesting to us are marked with an arrow ->.

_VNOP_CREATE: 1 push rbp 2 mov rbp, rsp 3 push r15 4 push r14 5 push r13 6 push r12 7 push rbx 8 sub rsp, 0x48 9 mov r15, r8 10 mov r12, rdx 11 mov r13, rsi -> 12 mov rbx, rdi 13 lea rax, qword [___stack_chk_guard] 14 mov rax, qword [rax] 15 mov qword [rbp+-48], rax -> 16 lea rax, qword [_vnop_create_desc] ; _vnop_create_desc 17 mov qword [rbp+-112], rax 18 mov qword [rbp+-104], rdi 19 mov qword [rbp+-96], rsi 20 mov qword [rbp+-88], rdx 21 mov qword [rbp+-80], rcx 22 mov qword [rbp+-72], r8 -> 23 mov rax, qword [rdi+0xd0] -> 24 movsxd rcx, dword [_vnop_create_desc] 25 lea rdi, qword [rbp+-112] -> 26 call qword [rax+rcx*8] 27 mov r14d, eax 28 test eax, eax ….

errno_t
VNOP_CREATE(vnode_t dvp, vnode_t * vpp, struct componentname * cnp, struct vnode_attr * vap, vfs_context_t ctx)
{
  int _err;
  struct vnop_create_args a;

  a.a_desc = &vnop;_create_desc; a.a_dvp = dvp; a.a_vpp = vpp;
  a.a_cnp = cnp; a.a_vap = vap; a.a_context = ctx;

  _err = (*dvp->v_op[vnop_create_desc.vdesc_offset])(&a;);
…

We will scan the assembler instructions to find the shift in the vnode dvp. The “purpose” of assembler code is to call a function from the v_op table. To do this, the processor must follow these steps:

Upload dvp to register
Dereferencing it to get v_op (line 23)
Get vnop_create_desc.vdesc_offset (line 24)
Call a function (line 26)

If everything is clear with steps 2-4, then difficulties arise with the first step. How to understand which register dvp was loaded into? To do this, we used a method of emulating a function that monitors the movements of the desired pointer. According to the System V x86_64 calling convention, the first argument is passed in the rdi register. Therefore, we decided to keep track of all the registers that contain rdi. In my example, these are the rbx and rdi registers. Also, a copy of the register can be saved on the stack, which is found in the debug version of the kernel.

Knowing that the rbx and rdi registers store dvp, we find out that line 23 dereferenced vnode to get v_op. So we get the assumption that the displacement in the structure is 0xd0. To confirm the correct decision, we continue to scan and make sure that the function is called correctly (lines 24 and 26).

This method is safer, but, unfortunately, it also has disadvantages. We have to rely on the fact that the pattern of the function (namely the 4 steps that we talked about above) will be the same. However, the probability of changing the pattern of the function is an order of magnitude less than the probability of changing the order of the fields. So we decided to stop on the second method.

Replace the pointers in the table

After finding v_op, the question arises, how to use this pointer? There are two different ways - overwrite the function in the table (third arrow in the picture) or overwrite the table in vnode (second arrow in the picture).

At first it seems that the first option is more profitable, because we just need to replace one pointer. However, this approach has 2 significant drawbacks. Firstly, the v_op table is the same for all vnode of a given file system (v_op for HFS +, v_op for APFS, ...), so filtering by vnode is required, which can be very expensive - you will have to filter out extra vnode on each write operation. Secondly, the table is written on the Read-Only page. This limitation can be circumvented if you use recording via IOMappedWrite64, bypassing system checks. Also, if kext with the file system driver is shipped, it will be difficult to figure out how to remove the patch.

The second option turns out to be more targeted and safe - the interceptor will be called only for the necessary vnode, and vnode memory initially allows Read-Write operations. Since the entire table is being replaced, it is necessary to allocate a little more memory (80 functions instead of one). And since the number of tables is usually equal to the number of file systems, the memory limit is completely negligible.

That is why kext uses the second method, although, I repeat, at first glance it seems that this option is worse.

As a result, our driver works as follows:

KAUTH API provides vnode
We are replacing the vnode table. If required, we intercept operations only for “interesting” vnode, for example, user documents
When intercepting, we check which process is recording, we filter out “our”
We send a synchronous UserSpace request to the client, who decides what exactly needs to be saved.

What happened

Today we have a ready-made module, which is an extension of the macOS kernel. It is used in Acronis products with Active Protection technology. The service is installed with the program and adds our driver to the system. Thanks to this, the backup takes place in real time and takes into account any changes to the file system at the granular level.