Protecting Backups with ChatGPT4 — Building a LastHope Server

Situation – two Hyper-V servers, a dozen virtual machines, VBR CE installed on each Hyper-V, backups stored on adjacent disks + once a week directories with backups are synchronized with an old NetGear NAS.

It would seem – what could go wrong here??!! Everything! Despite the fact that Hyper-V is not in a domain, it is quite possible to break it over the network if an intruder gets to at least one workstation (or connects via WiFi). NetGear NAS is already outdated, the firmware contains known vulnerabilities. In general, backups are not protected and in the event of an attack from the inside – the chances of getting problems are quite high.

Solution – we make a separate hacker-protected NAS based on Debian Linux 12 + iptables + scripts from ChatGPT4.

Composition of a homemade NAS

We take 2 disks of 4 TB each, install Linux Debian 12, and organize the disks according to the diagram using the standard CLI-TUI installer:

NAME          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda             8:0    0  3,6T  0 disk
├─sda1          8:1    0  1,9G  0 part  /boot/efi
├─sda2          8:2    0 14,9G  0 part  [SWAP]
├─sda3          8:3    0 74,5G  0 part
│ └─md1         9:1    0 74,4G  0 raid1 /
└─sda4          8:4    0  3,5T  0 part
  └─vg1-disk2 253:0    0  3,5T  0 lvm   /backup2
sdb             8:16   0  3,6T  0 disk
├─sdb1          8:17   0  1,9G  0 part
├─sdb2          8:18   0 14,9G  0 part  [SWAP]
├─sdb3          8:19   0 74,5G  0 part
│ └─md1         9:1    0 74,4G  0 raid1 /
└─sdb4          8:20   0  3,5T  0 part
  └─vg0-disk1 253:1    0  3,5T  0 lvm   /backup1

Additionally, we check the boot from each disk (for example, when booting the PC via F8).

The number of cores and RAM is not very important, the network is 1 Gbit/sec.

Additional programs (relative to minimal Linux)

  • cifs-utils – needed for automatic mounting of SMB shares with backups on NAS

  • iptables-persistent – script for loading firewall rules from /etc/iptables/rules.v4 file at OS startup

Automatic mounting of shares with backups

In the file /etc/fstab we write the share address:

//192.168.XXX.XXX/backup    /mnt/nas164    cifs    vers=1.0,credentials=/etc/keys164.fstab     0       0

In the file /etc/keys164.fstab we write the credentials for authentication:

username=veeam
password=XXXXXXXXXXXXXXXX

We check that everything connects and mounts when we reboot:

Файловая система          1K-блоков Использовано   Доступно Использовано% Cмонтировано в
udev                        3960532            0    3960532            0% /dev
tmpfs                        797040          952     796088            1% /run
/dev/md1                   76255012      6106432   69351612            9% /
tmpfs                       3985192            0    3985192            0% /dev/shm
tmpfs                          5120            0       5120            0% /run/lock
/dev/mapper/vg1-disk2    3750348432   2720716580  991502352           74% /backup2
/dev/mapper/vg0-disk1    3750348432   1820983080 1891235852           50% /backup1
/dev/sda1                   1947916         5972    1941944            1% /boot/efi
//192.168.200.164/backup 5812666536   1648055344 4164611192           29% /mnt/nas164
tmpfs                        797036            0     797036            0% /run/user/0

Statement of the task for ChatGPT4

Hooray – we are ready to start setting tasks for the neural network:

  • Condition 1 – only Full Backup files from Veeam (extension .vbk) are extracted from NAS on a schedule and placed in the /backup2 and /backup1 directories

  • Condition 2 – first the files must be copied to /backup2, while we need no more than 3 Full Backup files of each object (virtual machine)

  • Condition 3 – if there are more than 3 files on /backup2 – the oldest backup is moved to /backup1

  • Condition 4 – if there are more than 2 files on /backup1 – delete the oldest backup

  • Condition 5 – all operations must check for free disk space and the script must not try to fill the disk in a situation where this is a clearly failed idea (there is no space)

  • Condition 6 – after executing the script, we want to receive a report in Telegram to see which files were not touched, which files were copied from the NAS, and which files were moved from /backup2 to /backup1

We write a similar technical specification to the neural network (there was more than one request – there were many clarifications in the process) – as a result we get the following results:

The main script that will implement the conditions described above:

#!/bin/bash

# Проверяем, что передано ровно три аргумента
if [ $# -ne 3 ]; then
  echo "Ошибка: скрипту необходимо передать три каталога - исходный, каталог назначения и архивный каталог"
  exit 1
fi

# Сохраняем аргументы в переменные
# Исходный каталог, где нужно искать файлы .vbk
SOURCE_DIR="$1"
# Каталог назначения на другом диске
DEST_DIR="$2"
# Архивный диск - для самых самых старых бекапов
ARCHIVE_DIR="$3"

# Максимальное количество файлов .vbk в каталоге назначения и архиве
MAX_FILES=2

# Проверка наличия каталогов назначения и архива
if [ ! -d "$DEST_DIR" ] || [ ! -d "$ARCHIVE_DIR" ]; then
  echo "Каталог назначения или архивный каталог не существует"
  exit 1
fi

# Функция для получения свободного места на диске в байтах
get_free_space() {
  df -B1 "$1" | tail -1 | awk '{print $4}'
}

# Функция для удаления самого старого файла, если количество файлов в архиве превышает MAX_FILES
cleanup_archive() {
  local count=$(ls -1 "$ARCHIVE_DIR"/*.vbk 2>/dev/null | wc -l)
  if [ "$count" -gt "$MAX_FILES" ]; then
    oldest_file=$(ls -1t "$ARCHIVE_DIR"/*.vbk | tail -1)
    echo "Удаление самого старого файла в архиве: $oldest_file"
    rm "$oldest_file"
  fi
}

# Функция для перемещения самого старого файла в архив
move_oldest_file_to_archive() {
  local count=$(ls -1 "$DEST_DIR"/*.vbk 2>/dev/null | wc -l)
  if [ "$count" -gt "$MAX_FILES" ]; then
    oldest_file=$(ls -1t "$DEST_DIR"/*.vbk | tail -1)
    filesize=$(stat -c%s "$oldest_file")
    free_space=$(get_free_space "$ARCHIVE_DIR")
    if [ "$free_space" -ge "$filesize" ]; then
      echo "Перемещение самого старого файла в архив: $oldest_file"
      mv "$oldest_file" "$ARCHIVE_DIR"
      cleanup_archive
    else
      echo "Недостаточно места в архиве для перемещения файла $oldest_file"
    fi
  fi
}

# Поиск и копирование файлов
for file in "$SOURCE_DIR"/*.vbk; do
  # Проверка существования файлов .vbk в исходном каталоге
  if [ ! -e "$file" ]; then
    echo "Нет файлов .vbk в каталоге: $SOURCE_DIR"
    exit 1
  fi

  # Получение имени файла
  filename=$(basename "$file")

  # Проверка существования файла в каталоге назначения
  if [ ! -e "$DEST_DIR/$filename" ]; then
    # Размер файла в байтах
    filesize=$(stat -c%s "$file")

    # Свободное место в каталоге назначения
    free_space=$(get_free_space "$DEST_DIR")

    # Проверка достаточности свободного места
    if [ "$free_space" -ge "$filesize" ]; then
      # Перемещение самого старого файла в архив, если файлов больше 2
      move_oldest_file_to_archive
      echo "Копирование файла $filename в $DEST_DIR"
      cp "$file" "$DEST_DIR"
    else
      echo "Недостаточно места для копирования файла $filename. Требуется: $filesize байт, доступно: $free_space байт"
    fi
  else
    echo "Файл $filename уже существует в $DEST_DIR"
  fi
done

Next, we test the script, debug it in different situations. Then we write the second script ourselves, which will consistently perform the described actions for all directories of all backups (I remind you, we have 10 virtual machines):

#!/bin/bash

/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/DC2/            /backup2/HYPERV2/DC2/            /backup1/HYPERV2/DC2/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/KSC14/          /backup2/HYPERV2/KSC14/          /backup1/HYPERV2/KSC14/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/DC1/             /backup2/HYPERV/DC1/             /backup1/HYPERV/DC1/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/ICINGA/          /backup2/HYPERV/ICINGA/          /backup1/HYPERV/ICINGA/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/SUPPORT/         /backup2/HYPERV/SUPPORT/         /backup1/HYPERV/SUPPORT/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/1C-LICENSE/      /backup2/HYPERV/1C-LICENSE/      /backup1/HYPERV/1C-LICENSE/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/1C-SRV2/         /backup2/HYPERV/1C-SRV2/         /backup1/HYPERV/1C-SRV2/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/MAILSERVER/     /backup2/HYPERV2/MAILSERVER/     /backup1/HYPERV2/MAILSERVER/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/DIRSERVER/       /backup2/HYPERV/DIRSERVER/       /backup1/HYPERV/DIRSERVER/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/FILESERVER/     /backup2/HYPERV2/FILESERVER/     /backup1/HYPERV2/FILESERVER/

We ask another question in ChatGPT4 (about the script that will send us a file in Telegram) – we get a third script:

#!/bin/bash

# Запускаем проверку наличия и сравнения бекапов:
bash /root/Scripts/all-check.sh > /root/Scripts/backup-rotate-history.txt

# Замените 'TELEGRAM_BOT_TOKEN' на токен своего бота
TOKEN="41XXXXXXXXX:AAF7LKAw4iX0hwXhaUjXXXXXXXXXXXXXXXX"

# Замените 'CHAT_ID' на ID нужного чата
CHAT_ID="-19XXXXXXXX"

BACKUP_LIST="/root/Scripts/backup-rotate-history.txt"

if [ -n "$(cat /root/Scripts/backup-rotate-history.txt)" ]; then
  curl -F chat_id=$CHAT_ID -F text="COMPANY-BCK-HISTORY" -F filename=backup-rotate-history.txt -F document=@$BACKUP_LIST https://api.telegram.org/bot$TOKEN/sendDocument
fi

All that remains is to test everything again on test sets of files and hang the third script in /etc/crontab:

30 10   1 * *   root    /root/Scripts/telegram-notification.sh

Protecting Debian Linux NAS from Intrusion

In the current scheme on Debian, only one port is open – SSH (TCP/22). Using iptables and iptables-persistent, fill the file /etc/iptables/rules.v4 like this:

root@lasthope:~/Scripts# cat /etc/iptables/rules.v4
# Generated by iptables-save v1.8.9 (nf_tables) on Tue Apr 16 14:39:19 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -s XX.XX.XX.XX/32 -i enp3s0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i enp3s0 -p tcp -m tcp --dport 22 -j DROP
COMMIT
# Completed on Tue Apr 16 14:39:19 2024

In principle, if you want even more paranoia – you can install KnockD, close SSH altogether. You can also even add OneTime sequences for Port Knock. Or even attach F2A to SSH via phone…

We also protect the IP address from which we will log in via SSH and check the server – a separate PC with Linux, no open ports at all, iptables blocks all incoming connections.

Naturally, it is worth checking this server (Linux NAS) periodically, updating packages (hello, recent vulnerability in OpenSSH), checking the backups themselves (once a quarter we download VBR, try to deploy).

If you want even more protection, buy a USB HDD 3.0, download the archive, put it in a safe…

P.S.

Backup is a regular, repeatable process – you can't set it up once and not monitor, check, test, and improve it.

Either you deal with it, or this situation will deal with you – when there is hacking and encryption (with subsequent ransom extortion), or hardware failures, personnel errors, etc.

Good luck to everyone and I look forward to your solutions and suggestions for protecting backups in the comments!

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *