Catching and handling events in the Linux file system

11 min


Introduction

In the previous article, we looked at building and installing the package on Linux systems, in which we mentioned the Linux Kernel Module (LKM) and promised to reveal later details about the path to it and its creation. Well, his time has come. LKM – we choose you.

Need for implementation

“We replaced the Windows driver with the Linux Kernel Module LKM …” so, let’s go back mentally to the very beginning of the path. We have a Windows driver that monitors and intercepts file access events. How to port it or what to replace it in Linux systems? Digging into architectureafter reading about interception and the implementation of similar technologies in Linux – we realized that the task is absolutely non-trivial, containing a bunch of pitfalls.

Inotify

Throwing fishing rods on a couple of forums, after consulting with colleagues, it was decided to “dig” aside Inotify… Inotify is a file monitor that logs events in the system after they have occurred. But he has a “brother” – fanotify… In it, we can add an accessibility restriction for opening and copying events file… But we need to have the same ability for delete, rename, move events, and therefore fanotify will not help us with this. I want to note that fanotify is a userspace utility, so when using it there are no problems with platform portability.

Virtual File System

The next stage of the study was the possibility of intercepting requests using VFS

After analysis VFS based Dtrace, eBPF and bcc, it became clear that when using this technology it is possible to monitor events occurring in the system. In this case, the interception is carried out through LKM… During the study of the implementation of various modules for different kernels, the following was revealed: • interception does not always allow tracing the full path to a file; • when intercepting access to a file through an open application, and not from the explorer, there is no path to the file in the arguments; • each core needs its own implementation.

Janus, SElinux and AppArmor

During the research, an article was found on expanding the functionality of the kernel security system Linux… It follows that there are enough solutions on the market. The most easily implemented is Janus… The disadvantage of the solution is the lack of support for fresh kernels and all the above-described problems of the LKM hook. Implementation SELinux and AppArmor represents the quintessence of everything described and studied earlier. SELinux module includes main components: • security server; • Access Vector Cache (AVC); • tables of network interfaces; • network notification signal code; • its own virtual file system (selinuxfs) and implementation of interceptor functions.

The long-awaited decision

After all these endless “but”, Habr came to our aid! Having stumbled upon the article, it became clear that this was our case.

Intercept handling

After examining the suggested data on ftrace and implementation from the article itself, made a similar LKM module based on ftrace. This utility, in turn, works on the basis of the debugfs file system, which is mounted by default in most modern Linux distributions. Hooks added to events to the already existing clone and open: • openat, • rename, • unlink, • unlinkat. Thus, it was possible to handle opening, renaming, moving, copying, deleting a file.

Interaction

Now we need to implement communication between the core module and the userspace application. There are different approaches to solving this problem, but basically there are two: • socket between kernel and userspace; • writing / reading in the system directory to a file.

As a result, we chose netlink socket, since on Windows we use a similar interface – FltSendMessage… Could use inet socket, but this is the least secure solution. Also faced such a problem that on .Net Core, on which the userspace application is implemented, is absent realization netlink.

Therefore, I had to implement a dynamic library with the implementation netlink and already connect it to the project.


int open_netlink_connection(void)
{
    //initialize our variables
    int sock;
    struct sockaddr_nl addr;
    int group = NETLINK_GROUP;

    //open a new socket connection
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_USERSOCK);

    //if the socket failed to open,
    if (sock < 0) 
    {
        //inform the user
        printf("Socket failed to initialize.n");
        //return the error value
        return sock;
    }

    //initialize our addr structure by filling it with zeros
    memset((void *) &addr, 0, sizeof(addr));
    //specify the protocol family
    addr.nl_family = AF_NETLINK;
    //set the process id to the current process id
    addr.nl_pid = getpid();

    //bind the address to the socket created, and if it failed,
    if (bind(sock, (struct sockaddr *) &addr, sizeof(addr)) < 0) 
    {
        //inform the user
        printf("bind < 0.n");
        //return the function with a symbolic error code
        return -1;
    }

    //set the option so that we can receive packets whose destination
    //is the group address specified (so that we can receive the message broadcasted by the kernel)
    if (setsockopt(sock, 270, NETLINK_ADD_MEMBERSHIP, &group, sizeof(group)) < 0) 
    {
        //if it failed, inform the user
        printf("setsockopt < 0n");
        //return the function with a symbolic error code
        return -1;
    }

    //if we got thus far, then everything
    //went fine. Return our socket.
    return sock;
}

char* read_kernel_message(int sock)
{
    //initialize the variables
    //that we are going to need
    struct sockaddr_nl nladdr;
    struct msghdr msg;
    struct iovec iov;
    char* buffer[CHUNK_SIZE];
    char* kernelMessage;
    int ret;

    memset(&msg, 0, CMSG_SPACE(MAX_PAYLOAD));
    memset(&nladdr, 0, sizeof(nladdr));
    memset(&iov, 0, sizeof(iov));
    //specify the buffer to save the message
    iov.iov_base = (void *) &buffer;
    //specify the length of our buffer
    iov.iov_len = sizeof(buffer);

    //pass the pointer of our sockaddr structure
    //that will save the source IP and port of the connection
    msg.msg_name = (void *) &(dest_addr);
    //give the size of our structure
    msg.msg_namelen = sizeof(dest_addr);
    //pass our scatter/gather I/O structure pointer
    msg.msg_iov = &iov;
    //we will pass only one buffer array,
    //therefore we will specify that here
    msg.msg_iovlen = 1;

    //listen/wait for new data
    ret = recvmsg(sock, &msg, 0);

    //if message was received successfully,
    if(ret >= 0)
    {
        //get the string data and save them to a local variable
        char* buf = NLMSG_DATA((struct nlmsghdr *) &buffer);

        //allocate memory for our kernel message
        kernelMessage = (char*)malloc(CHUNK_SIZE);

        //copy the kernel data to our allocated space
        strcpy(kernelMessage, buf);

        //return the pointer that points to the kernel data
        return kernelMessage;
    }
    
    //if we got that far, reading the message failed,
    //so we inform the user and return a NULL pointer
    printf("Message could not received.n");
    return NULL;
}

int send_kernel_message(int sock, char* kernelMessage)
{
    //initialize the variables
    //that we are going to need
    struct msghdr msg;
    struct iovec iov;
    char* buffer[CHUNK_SIZE];    
    int ret;

    memset(&msg, 0, CMSG_SPACE(MAX_PAYLOAD));
    memset(&iov, 0, sizeof(iov));

    nlh = (struct nlmsghdr *)malloc(NLMSG_SPACE(MAX_PAYLOAD));
    memset(nlh, 0, NLMSG_SPACE(MAX_PAYLOAD));
    nlh->nlmsg_len = NLMSG_SPACE(MAX_PAYLOAD);
    nlh->nlmsg_pid = getpid();
    nlh->nlmsg_flags = 0;

    char buff[160];
    snprintf(buff, sizeof(buff), "From:DSSAgent;Action:return;Message:%s;", kernelMessage);
    strcpy(NLMSG_DATA(nlh), buff);

    iov.iov_base = (void *)nlh;
    iov.iov_len = nlh->nlmsg_len;
    //pass the pointer of our sockaddr structure
    //that will save the source IP and port of the connection
    msg.msg_name = (void *) &(dest_addr);
    //give the size of our structure
    msg.msg_namelen = sizeof(dest_addr);
    msg.msg_iov = &iov;
    msg.msg_iovlen = 1;
    printf("Sending message to kernel (%s)n",(char *)NLMSG_DATA(nlh));
    ret = sendmsg(sock, &msg, 0);
    return ret;
}

int sock_netlink_connection()
{
	sock_fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_USER);
    if (sock_fd < 0)
        return -1;


    memset(&src_addr, 0, sizeof(src_addr));
    src_addr.nl_family = AF_NETLINK;
    src_addr.nl_pid = getpid(); /* self pid */


    bind(sock_fd, (struct sockaddr *)&src_addr, sizeof(src_addr));


    memset(&dest_addr, 0, sizeof(dest_addr));
    dest_addr.nl_family = AF_NETLINK;
    dest_addr.nl_pid = 0; /* For Linux Kernel */
    dest_addr.nl_groups = 0; /* unicast */


    nlh = (struct nlmsghdr *)malloc(NLMSG_SPACE(MAX_PAYLOAD));
    memset(nlh, 0, NLMSG_SPACE(MAX_PAYLOAD));
    nlh->nlmsg_len = NLMSG_SPACE(MAX_PAYLOAD);
    nlh->nlmsg_pid = getpid();
    nlh->nlmsg_flags = 0;


    strcpy(NLMSG_DATA(nlh), "From:DSSAgent;Action:hello;");


    iov.iov_base = (void *)nlh;
    iov.iov_len = nlh->nlmsg_len;
    msg.msg_name = (void *)&dest_addr;
    msg.msg_namelen = sizeof(dest_addr);
    msg.msg_iov = &iov;
    msg.msg_iovlen = 1;


    printf("Sending message to kerneln");
    sendmsg(sock_fd, &msg, 0);
    printf("Waiting for message from kerneln");


    /* Read message from kernel */
    recvmsg(sock_fd, &msg, 0);
    printf("Received message payload: %sn", (char *)NLMSG_DATA(nlh));
	
	return sock_fd;
}
void sock_netlink_disconnection(int sock)
{
	close(sock);
    free(nlh);
}

Also, later it turned out that some functions are missing in Net.Core – for example, searching by the process pid of the username that owns this processExamples given implementation turned out weight, but, within the framework of our application, it was not possible to implement them. Therefore, in the same library, we have implemented our function for finding the user’s uid, by which the name can be found using system functions.

char* get_username_by_pid(int pid)
{ 
  register struct passwd *pw;
  register uid_t uid;
  int c;
  FILE *fp;
  char filename[255];
  sprintf(filename, "/proc/%d/loginuid", pid);
  char cc[8];
  
  // чтение из файла
  if((fp= fopen(filename, "r"))==NULL)
    {
        perror("Error occured while opening file");
        return "";
    }
  // считываем, пока не дойдем до конца
  while((fgets(cc, 8, fp))!=NULL) {}
     
  fclose(fp);
  
  uid = atoi(cc);

  pw = getpwuid (uid);
  if (pw)
  {
      return pw->pw_name;
  }
  else
  {
      return "";
  }
}

Modification of the module

As a result, we added a netlink connection to the LKM initialization.

static int fh_init(void)
{
    int err;
	struct netlink_kernel_cfg cfg =
	{
#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 6, 0)
		.groups = 1,
#endif
		.input = nl_recv_msg,
	};

#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 36)
	nl_sk = netlink_kernel_create(&init_net, NETLINK_USER, &cfg);
#elif LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 32)
	nl_sk = netlink_kernel_create(&init_net, NETLINK_USER, 0, nl_recv_msg, NULL, THIS_MODULE);
#else
	nl_sk = netlink_kernel_create(NETLINK_USER, 0, nl_recv_msg, THIS_MODULE);
#endif

	if (!nl_sk)
	{
		printk(KERN_ERR "%s Could not create netlink socketn", __func__);
		return 1;
	}

	err = fh_install_hooks(hooks, ARRAY_SIZE(hooks));
	if (err)
		return err;

	p_list_hook_files = (tNode *)kmalloc(sizeof(tNode), GFP_KERNEL);
	p_list_hook_files->next = NULL;
	p_list_hook_files->value = 0;

	pr_info("module loadedn");

	return 0;
}
module_init(fh_init);

static void fh_exit(void)
{
	delete_list(p_list_hook_files);
	fh_remove_hooks(hooks, ARRAY_SIZE(hooks));
	netlink_kernel_release(nl_sk);
	pr_info("module unloadedn");
}
module_exit(fh_exit);

The Socket is waiting to catch the file access event. The module, intercepting the event, passes the file name, pid and process name. Userspace application, receiving this information, processes it and answers what to do with the file (block or allow access). Subsequently, the module returns the appropriate system call.

static void send_msg_to_user(const char *msgText)
{
	int msgLen = strlen(msgText);

	struct sk_buff *skb = nlmsg_new(NLMSG_ALIGN(msgLen), GFP_KERNEL);

	if (!skb)
	{
		printk(KERN_ERR "%s Allocation skb failure.n", __func__);
		return;
	}

	struct nlmsghdr *nlh = nlmsg_put(skb, 0, 1, NLMSG_DONE, msgLen, 0);

	if (!nlh)
	{
		printk(KERN_ERR "%s Create nlh failure.n", __func__);
		nlmsg_free(skb);
		return;
	}

	NETLINK_CB(skb).dst_group = 0;
	strncpy(nlmsg_data(nlh), msgText, msgLen);

	int errorVal = nlmsg_unicast(nl_sk, skb, pid);

	if (errorVal < 0)
		printk(KERN_ERR "%s nlmsg_unicast() error: %dn", __func__, errorVal);
}

static void return_msg_to_user(struct nlmsghdr *nlh)
{
	pid = nlh->nlmsg_pid;

	const char *msg = "Init socket from kernel";
	const int msg_size = strlen(msg);

	struct sk_buff *skb = nlmsg_new(msg_size, 0);
	if (!skb)
	{
		printk(KERN_ERR "%s Failed to allocate new skbn", __func__);
		return;
	}

	nlh = nlmsg_put(skb, 0, 0, NLMSG_DONE, msg_size, 0);
	NETLINK_CB(skb).dst_group = 0;
	strncpy(nlmsg_data(nlh), msg, msg_size);

	int res = nlmsg_unicast(nl_sk, skb, pid);
	if (res < 0)
		printk(KERN_ERR "%s Error while sending back to user (%i)n", __func__, res);
}

Later, for the operation of another application in this module, we added the ability to block access to a specific file (by its full path) for all processes except a certain one (determined by the process pid).

static void parse_return_from_user(char *return_msg)
{
	char *msg = np_extract_value(return_msg, "Message", ';');
	const char *file_name = strsep(&msg, "|");

	printk(KERN_INFO "%s Name:(%s) Permiss:(%s)n", __func__, file_name, msg);

	if (strstr(msg, "Deny"))
		reload_name_list(p_list_hook_files, file_name, Deny);
	else
		reload_name_list(p_list_hook_files, file_name, Allow);
}

static void free_guards(void)
{
	// Possibly unpredictable behavior during cleaning
	memset(&guards, 0, sizeof(struct process_guards));
}

static void change_guards(char *msg)
{
	char *path = np_extract_value(msg, "Path", ';');
	char *count_str = np_extract_value(msg, "Count", ';');

	if (path && strlen(path) && count_str && strlen(count_str))
	{
		int i, found = -1;

		for (i = 0; i < guards.count; ++i)
			if (guards.process[i].file_path && !strcmp(path, guards.process[i].file_path))
				found = i;

		guards.is_busy = 1;

		int count;
		kstrtoint(count_str, 10, &count);

		if (count > 0)
		{
			if (found == -1)
			{
				strcpy(guards.process[guards.count].file_path, path);
				found = guards.count;
				guards.count++;
			}

			for (i = 0; i < count; ++i)
			{
				char buff[8];
				snprintf(buff, sizeof(buff), "Pid%d", i + 1);
				char *pid = np_extract_value(msg, buff, ';');
				if (pid && strlen(pid))
					kstrtoint(pid, 10, &guards.process[found].allow_pids[i]);
				else
					guards.process[found].allow_pids[i] = 0;
			}

			guards.process[found].allow_pids[count] = 0;
		}
		else
		{
			if (found >= 0)
			{
				for (i = found; i < guards.count - 1; ++i)
					guards.process[i] = guards.process[i + 1];

				guards.count--;
			}
		}

		guards.is_busy = 0;
	}
}

// Example message is "From:CryptoCli;Action:clear;" or "From:DSSAgent;Action:init;"
static void nl_recv_msg(struct sk_buff *skb)
{
	printk(KERN_INFO "%s <--n", __func__);

	struct nlmsghdr *nlh = (struct nlmsghdr *)skb->data;

	printk(KERN_INFO "%s Netlink received msg payload:%sn", __func__, (char *)nlmsg_data(nlh));

	char *msg = (char *)nlmsg_data(nlh);

	if (msg && strlen(msg))
	{
		char *from = np_extract_value(msg, "From", ';');
		char *action = np_extract_value(msg, "Action", ';');

		if (from && strlen(from) && action && strlen(action))
		{
			
			if (!strcmp(from, "DSSAgent"))
			{
				if (!strcmp(action, "init"))
				{
					return_msg_to_user(nlh);
				}
				else if (!strcmp(action, "return"))
				{
					parse_return_from_user(msg);
				}
				else
				{
					printk(KERN_ERR "%s Failed msg, "From" is %s and "Action" is %sn", __func__, from, action);
				}
			}
			else if (!strcmp(from, "CryptoCli"))
			{
				if (!strcmp(action, "clear"))
				{
					free_guards();
				}
				else if (!strcmp(action, "change"))
				{
					change_guards(msg);
				}
				else
				{
					printk(KERN_ERR "%s Failed msg, "From" is %s and "Action" is %sn", __func__, from, action);
				}
			}
			else
			{
				printk(KERN_ERR "%s Failed msg, "From" is %s and "Action" is %sn", __func__, from, action);
			}

		}
		else
		{
			printk(KERN_ERR "%s Failed parse msg, don`t found "From" and "Action" (%s)n", __func__, msg);
		}
	}
	else
	{
		printk(KERN_ERR "%s Failed parse struct nlmsg_data, msg is emptyn", __func__);
	}

	printk(KERN_INFO "%s -->n", __func__);
}

static bool check_file_access(char *fname, int processPid)
{
	if (fname && strlen(fname))
	{
		int i;

		for (i = 0; i < guards.count; ++i)
		{
			if (!strcmp(fname, guards.process[i].file_path) && guards.process[i].allow_pids[0] != 0)
			{
				int j;
				
				for (j = 0; guards.process[i].allow_pids[j] != 0; ++j)
					if (processPid == guards.process[i].allow_pids[j])
						return true;

				return false;
			}
		}
		
		// Not found filename in guards
		if (strstr(fname, filetype))
		{
			char *processName = current->comm;

			printk(KERN_INFO "%s service pid = %dn", __func__, pid);
			printk(KERN_INFO "%s file name = %s, process pid: %d, , process name = %sn", __func__, fname, processPid, processName);

			if (processPid == pid)
			{
				return true;
			}
			else
			{
				add_list(p_list_hook_files, processPid, fname, None);

				char *buffer = kmalloc(4096, GFP_KERNEL);
				sprintf(buffer, "%s|%s|%d", fname, processName, processPid);
				send_msg_to_user(buffer);
				kfree(buffer);

				ssleep(5);

				bool ret = true;

				if (find_list(p_list_hook_files, fname) == Deny)
					ret = false;

				delete_node(p_list_hook_files, fname);

				return ret;
			}
		}
	}

	return true;
}

Integration into the installation process

Since the first two disadvantages of LKM were overcome through the implementation of ftrace, the third one has not been canceled. Not only is it necessary to assemble a module for each core, but already in the process of use it can “go bad”. It was decided to add its rebuild before each launch of the userspace application. In the article on building Linux packages, it was described that the “service” for which we implement handling of intercepting file access, we “daemonized” by adding to system. Therefore, for the .service daemon, we add two additional items, in addition to ExecStart and ExecStop:

ExecStartPre=/bin/sh /путь_до_расположения/prestart.sh
ExecStopPost=/sbin/rmmod имя_модуля.ko

and in prestart.sh itself:

#!/bin/sh

MOD_VAL=$(lsmod | grep имя_модуля | wc -l)

cd /путь_до_расположения_модуля
make clean
make all

if [ $MOD_VAL = 1 ]
then
    for proc in $(ps aux | grep DSS.Agent | awk '{print $2}'); do kill -9 $proc; done
else
    /sbin/insmod / путь_до_расположения_модуля/имя_модуля.ko
fi

Conclusion

In conclusion, I would like to note: perhaps the path we took was not the most “beautiful and elegant” one, but it contains a well-tried and tested logic of work on Windows OS. It would be useful to hear the opinion of the readers of the article in the comments. Perhaps there is a smarter solution to the problem. For example, our DevOps, at the moment when we were automating the build of the Linux package and processing / adding LKM, suggested implementing the logic using the Access Control List (ACL). Most likely, in the future we will deal with the processing of our product for Linux. And, yes, there will be a new article soon on how we ported MS Forms to Avalonia and its Linux integration.

Links that helped us

  • https://habr.com/ru/post/413241/ – article on creating LKM based on ftrace

  • https://superuser.com/questions/479746/how-to-find-pids-user-name-in-linux/479748 – where we can get the uid of the user in the system by the pid of the process

  • https://stackoverflow.com/questions/31653257/how-to-send-and-receive-messages-from-function-other-than-registered-callback-fu- netlink socket


0 Comments

Leave a Reply