Protecting Backups with ChatGPT4 — Building a LastHope Server
Situation – two Hyper-V servers, a dozen virtual machines, VBR CE installed on each Hyper-V, backups stored on adjacent disks + once a week directories with backups are synchronized with an old NetGear NAS.
It would seem – what could go wrong here??!! Everything! Despite the fact that Hyper-V is not in a domain, it is quite possible to break it over the network if an intruder gets to at least one workstation (or connects via WiFi). NetGear NAS is already outdated, the firmware contains known vulnerabilities. In general, backups are not protected and in the event of an attack from the inside – the chances of getting problems are quite high.
Solution – we make a separate hacker-protected NAS based on Debian Linux 12 + iptables + scripts from ChatGPT4.
Composition of a homemade NAS
We take 2 disks of 4 TB each, install Linux Debian 12, and organize the disks according to the diagram using the standard CLI-TUI installer:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 3,6T 0 disk
├─sda1 8:1 0 1,9G 0 part /boot/efi
├─sda2 8:2 0 14,9G 0 part [SWAP]
├─sda3 8:3 0 74,5G 0 part
│ └─md1 9:1 0 74,4G 0 raid1 /
└─sda4 8:4 0 3,5T 0 part
└─vg1-disk2 253:0 0 3,5T 0 lvm /backup2
sdb 8:16 0 3,6T 0 disk
├─sdb1 8:17 0 1,9G 0 part
├─sdb2 8:18 0 14,9G 0 part [SWAP]
├─sdb3 8:19 0 74,5G 0 part
│ └─md1 9:1 0 74,4G 0 raid1 /
└─sdb4 8:20 0 3,5T 0 part
└─vg0-disk1 253:1 0 3,5T 0 lvm /backup1Additionally, we check the boot from each disk (for example, when booting the PC via F8).
The number of cores and RAM is not very important, the network is 1 Gbit/sec.
Additional programs (relative to minimal Linux)
cifs-utils – needed for automatic mounting of SMB shares with backups on NAS
iptables-persistent – script for loading firewall rules from /etc/iptables/rules.v4 file at OS startup
Automatic mounting of shares with backups
In the file /etc/fstab we write the share address:
//192.168.XXX.XXX/backup /mnt/nas164 cifs vers=1.0,credentials=/etc/keys164.fstab 0 0In the file /etc/keys164.fstab we write the credentials for authentication:
username=veeam
password=XXXXXXXXXXXXXXXXWe check that everything connects and mounts when we reboot:
Файловая система 1K-блоков Использовано Доступно Использовано% Cмонтировано в
udev 3960532 0 3960532 0% /dev
tmpfs 797040 952 796088 1% /run
/dev/md1 76255012 6106432 69351612 9% /
tmpfs 3985192 0 3985192 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
/dev/mapper/vg1-disk2 3750348432 2720716580 991502352 74% /backup2
/dev/mapper/vg0-disk1 3750348432 1820983080 1891235852 50% /backup1
/dev/sda1 1947916 5972 1941944 1% /boot/efi
//192.168.200.164/backup 5812666536 1648055344 4164611192 29% /mnt/nas164
tmpfs 797036 0 797036 0% /run/user/0Statement of the task for ChatGPT4
Hooray – we are ready to start setting tasks for the neural network:
Condition 1 – only Full Backup files from Veeam (extension .vbk) are extracted from NAS on a schedule and placed in the /backup2 and /backup1 directories
Condition 2 – first the files must be copied to /backup2, while we need no more than 3 Full Backup files of each object (virtual machine)
Condition 3 – if there are more than 3 files on /backup2 – the oldest backup is moved to /backup1
Condition 4 – if there are more than 2 files on /backup1 – delete the oldest backup
Condition 5 – all operations must check for free disk space and the script must not try to fill the disk in a situation where this is a clearly failed idea (there is no space)
Condition 6 – after executing the script, we want to receive a report in Telegram to see which files were not touched, which files were copied from the NAS, and which files were moved from /backup2 to /backup1
We write a similar technical specification to the neural network (there was more than one request – there were many clarifications in the process) – as a result we get the following results:
The main script that will implement the conditions described above:
#!/bin/bash
# Проверяем, что передано ровно три аргумента
if [ $# -ne 3 ]; then
echo "Ошибка: скрипту необходимо передать три каталога - исходный, каталог назначения и архивный каталог"
exit 1
fi
# Сохраняем аргументы в переменные
# Исходный каталог, где нужно искать файлы .vbk
SOURCE_DIR="$1"
# Каталог назначения на другом диске
DEST_DIR="$2"
# Архивный диск - для самых самых старых бекапов
ARCHIVE_DIR="$3"
# Максимальное количество файлов .vbk в каталоге назначения и архиве
MAX_FILES=2
# Проверка наличия каталогов назначения и архива
if [ ! -d "$DEST_DIR" ] || [ ! -d "$ARCHIVE_DIR" ]; then
echo "Каталог назначения или архивный каталог не существует"
exit 1
fi
# Функция для получения свободного места на диске в байтах
get_free_space() {
df -B1 "$1" | tail -1 | awk '{print $4}'
}
# Функция для удаления самого старого файла, если количество файлов в архиве превышает MAX_FILES
cleanup_archive() {
local count=$(ls -1 "$ARCHIVE_DIR"/*.vbk 2>/dev/null | wc -l)
if [ "$count" -gt "$MAX_FILES" ]; then
oldest_file=$(ls -1t "$ARCHIVE_DIR"/*.vbk | tail -1)
echo "Удаление самого старого файла в архиве: $oldest_file"
rm "$oldest_file"
fi
}
# Функция для перемещения самого старого файла в архив
move_oldest_file_to_archive() {
local count=$(ls -1 "$DEST_DIR"/*.vbk 2>/dev/null | wc -l)
if [ "$count" -gt "$MAX_FILES" ]; then
oldest_file=$(ls -1t "$DEST_DIR"/*.vbk | tail -1)
filesize=$(stat -c%s "$oldest_file")
free_space=$(get_free_space "$ARCHIVE_DIR")
if [ "$free_space" -ge "$filesize" ]; then
echo "Перемещение самого старого файла в архив: $oldest_file"
mv "$oldest_file" "$ARCHIVE_DIR"
cleanup_archive
else
echo "Недостаточно места в архиве для перемещения файла $oldest_file"
fi
fi
}
# Поиск и копирование файлов
for file in "$SOURCE_DIR"/*.vbk; do
# Проверка существования файлов .vbk в исходном каталоге
if [ ! -e "$file" ]; then
echo "Нет файлов .vbk в каталоге: $SOURCE_DIR"
exit 1
fi
# Получение имени файла
filename=$(basename "$file")
# Проверка существования файла в каталоге назначения
if [ ! -e "$DEST_DIR/$filename" ]; then
# Размер файла в байтах
filesize=$(stat -c%s "$file")
# Свободное место в каталоге назначения
free_space=$(get_free_space "$DEST_DIR")
# Проверка достаточности свободного места
if [ "$free_space" -ge "$filesize" ]; then
# Перемещение самого старого файла в архив, если файлов больше 2
move_oldest_file_to_archive
echo "Копирование файла $filename в $DEST_DIR"
cp "$file" "$DEST_DIR"
else
echo "Недостаточно места для копирования файла $filename. Требуется: $filesize байт, доступно: $free_space байт"
fi
else
echo "Файл $filename уже существует в $DEST_DIR"
fi
doneNext, we test the script, debug it in different situations. Then we write the second script ourselves, which will consistently perform the described actions for all directories of all backups (I remind you, we have 10 virtual machines):
#!/bin/bash
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/DC2/ /backup2/HYPERV2/DC2/ /backup1/HYPERV2/DC2/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/KSC14/ /backup2/HYPERV2/KSC14/ /backup1/HYPERV2/KSC14/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/DC1/ /backup2/HYPERV/DC1/ /backup1/HYPERV/DC1/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/ICINGA/ /backup2/HYPERV/ICINGA/ /backup1/HYPERV/ICINGA/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/SUPPORT/ /backup2/HYPERV/SUPPORT/ /backup1/HYPERV/SUPPORT/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/1C-LICENSE/ /backup2/HYPERV/1C-LICENSE/ /backup1/HYPERV/1C-LICENSE/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/1C-SRV2/ /backup2/HYPERV/1C-SRV2/ /backup1/HYPERV/1C-SRV2/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/MAILSERVER/ /backup2/HYPERV2/MAILSERVER/ /backup1/HYPERV2/MAILSERVER/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV/DIRSERVER/ /backup2/HYPERV/DIRSERVER/ /backup1/HYPERV/DIRSERVER/
/root/Scripts/copy-full-backup-vbk-v3.sh /mnt/nas164/HYPERV2/FILESERVER/ /backup2/HYPERV2/FILESERVER/ /backup1/HYPERV2/FILESERVER/We ask another question in ChatGPT4 (about the script that will send us a file in Telegram) – we get a third script:
#!/bin/bash
# Запускаем проверку наличия и сравнения бекапов:
bash /root/Scripts/all-check.sh > /root/Scripts/backup-rotate-history.txt
# Замените 'TELEGRAM_BOT_TOKEN' на токен своего бота
TOKEN="41XXXXXXXXX:AAF7LKAw4iX0hwXhaUjXXXXXXXXXXXXXXXX"
# Замените 'CHAT_ID' на ID нужного чата
CHAT_ID="-19XXXXXXXX"
BACKUP_LIST="/root/Scripts/backup-rotate-history.txt"
if [ -n "$(cat /root/Scripts/backup-rotate-history.txt)" ]; then
curl -F chat_id=$CHAT_ID -F text="COMPANY-BCK-HISTORY" -F filename=backup-rotate-history.txt -F document=@$BACKUP_LIST https://api.telegram.org/bot$TOKEN/sendDocument
fiAll that remains is to test everything again on test sets of files and hang the third script in /etc/crontab:
30 10 1 * * root /root/Scripts/telegram-notification.shProtecting Debian Linux NAS from Intrusion
In the current scheme on Debian, only one port is open – SSH (TCP/22). Using iptables and iptables-persistent, fill the file /etc/iptables/rules.v4 like this:
root@lasthope:~/Scripts# cat /etc/iptables/rules.v4
# Generated by iptables-save v1.8.9 (nf_tables) on Tue Apr 16 14:39:19 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -s XX.XX.XX.XX/32 -i enp3s0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i enp3s0 -p tcp -m tcp --dport 22 -j DROP
COMMIT
# Completed on Tue Apr 16 14:39:19 2024In principle, if you want even more paranoia – you can install KnockD, close SSH altogether. You can also even add OneTime sequences for Port Knock. Or even attach F2A to SSH via phone…
We also protect the IP address from which we will log in via SSH and check the server – a separate PC with Linux, no open ports at all, iptables blocks all incoming connections.
Naturally, it is worth checking this server (Linux NAS) periodically, updating packages (hello, recent vulnerability in OpenSSH), checking the backups themselves (once a quarter we download VBR, try to deploy).
If you want even more protection, buy a USB HDD 3.0, download the archive, put it in a safe…
P.S.
Backup is a regular, repeatable process – you can't set it up once and not monitor, check, test, and improve it.
Either you deal with it, or this situation will deal with you – when there is hacking and encryption (with subsequent ransom extortion), or hardware failures, personnel errors, etc.
Good luck to everyone and I look forward to your solutions and suggestions for protecting backups in the comments!
