How to backup websites and servers?

This is a burning data center of the largest hosting provider in Europe – OVH. After the fire, some projects never recovered, others licked their wounds for a long time. And to prevent this from happening, you need to make regular backups. Today we will talk about how, with what frequency and what to include in them.

Everything that is written below is the experience of our company. I tried to write as little theory as possible and focus on practice and personal experience. If you have something to add or want to discuss – welcome in the comments or in a personal (contacts in the profile).

Why are backups needed?

The most obvious is the situation in the photo. Data centers often fail in recent years. No, this does not mean that they have become less reliable. It’s just that there are more of them and such cases often get into the news feeds. We also had more exotic options. For example, once we lost a client project because hosting owners quarreled with each other and one of them came to the data center, cut everything down and took away all the servers. As you probably guessed, we were not allowed to pick up the files. We quickly restored the project from the reserves.

Viruses, trojans, backdoors. Recently, this is the most common situation when we use backups. Especially against the backdrop of famous events. Recently, viruses often delete all files and the site in the state that it is cannot be restored. so you can’t do without a backup.

Developer error. For example, we did not take into account some points when creating synchronization and crashed additional fields for all products. Or, for example, “urgent” edits in a battle remove the status of some orders and the entire order synchronization service falls apart like a house of cards. In such cases, a quick rollback of the database from the reserve saves the situation. We are all human and we also have such cases, so there are always backups, even if one of the top developers works on the project.

The reserve makes it possible to quickly raise the project locally. Perhaps this is the most non-obvious advantage of a backup. A backup is always a packed site and this makes it possible to quickly download the necessary data and raise the project locally. On some projects, we do such manipulations automatically, wrapping it in Vagrant, which allows the developer to quickly raise a local copy of the site from the latest backup without unnecessary dancing with a tambourine. “Fast”, of course, is a relative concept, and if the developer went to a remote village, where he only has 3g Internet and he catches from a base station from a neighboring village, then he will raise it only when he retires.

Where to store backups?

There are many storage options. We settled on clouds for personal use (Yandex disk, cloud.mail.ru, dropbox and other similar services). Judge for yourself, the cost of a terabyte starts from 2000 rubles and there are no additional payments for traffic, connections, etc. We select the service based on the needs of the client, sometimes even a free account is enough.

Briefly about the alternatives:

• Backups from the host. Not only does the infrastructure for storing backups belong to the same company, they are also usually stored in the same data center as your main site, so in case of problems, you will lose both the site and backups.

• Clouds for business. These are all kinds of AWS and other Yandex clouds. Now only lazy people don’t do that. In general, it is expensive, very complex billing and, frankly, they are not for this.

• Server in another data center. Quite a working option for a large project. It loses to the clouds in terms of storage costs, but wins in flexibility and speed of creating backups. For example, on the second server, you can set up a replica of the database and make copies from it at any frequency. At the same time, you can keep an active mirror on this server.

• A place in the Bitrix cloud. Bitrix has a cloud backup and if you have an active license, then you have a tool and storage space (from 2 to 10 GB depending on the license). It is impossible to expand this place (I found out from support). There is frankly little space, backup and recovery only by standard means. There are a number of clients who use it, but it is difficult to put more than one backup there.

What and how to store?

In fact, the composition of copies is not so simple.

Database. We usually work with mysql and they almost always forget to include stored procedures in the backpap, since the mysqldump utility does not do this by default. Not every project has storage, but you need to remember this.

A copy of the database should be stored separately from the files, and not shoved into one archive with the site files, as is often done. If there are several databases on the server, then each database should be in a separate archive, because most often, developers “break” exactly the data and a copy of the database is needed more often than a copy of the files.

Files. If there are many sites on the hosting/server, then each site should be in a separate archive (including separate archives for dev and stage servers). For online stores and various kinds of fashion sites, it is better to separate the media (pictures, videos) into a separate archive, because. they are less often needed for recovery, and in the event of an accident, the archive with the code will download faster in this case.

cron scripts. On complex projects in the crown, as a rule, there is more than one call. And if the server is lost, restoring all cron scripts is very difficult, and sometimes impossible. Therefore, cron should always be included in backups. Yes, we know what crunz and its analogues are, but ready-made projects often come to us for support and we have to work with what we have, and only then refactor.

Other scripts and settings. Most often, these are configs for supervisord and sphinx. It is enough to backup them once a month, or keep a copy in the git.

The frequency of backups depends on the project. Our standard scheme is 7 daily database copies and weekly file backups for 60 days. We also add the database to the weekly copies of the files (the database and file archives, even in this case, are stored separately according to the recommendations above). Such a scheme is sufficient for basic recovery scenarios. We create backup copies during hours of minimum load on the server, as a rule, this is one in the morning according to server time (you need to take into account the time zone of the main audience of the project).

Backups are always created before deployment and in any situation where there is a high probability of dropping the client’s project, for example, before updating Bitrix. Such backups are created at the request of the project manager or developer.

How to do?

For the company’s clients, we use our own scripts that do all the things described above and upload a copy to the prepared storage. At the same time, notifications are sent to telegrams with service information and backup status. The solution is only suitable for servers and VDS. You can’t run it on regular hosting. In general, there is nothing complicated in our scripts and the backbone of the solution is googled on the request “webdav backups for free and without SMS”.

For popular CMS, there are separate solutions that do almost the same thing. For example, for Bitrix we sometimes use ammina.backup (not advertising). In general, the module works stably, but not on every hosting. It does not work, as a rule, due to hosting restrictions on consumed resources. Changing the tariff plan to a more expensive one removes the restriction and makes it work.

There are also similar solutions for other CMS, but we didn’t have experience working with them, so if you throw working and proven options in the comments, I’ll be grateful.

Checked and forgot?

Another moment that everyone forgets. Backups need to be verified. We do this regularly once a week for each client.

The main problems that arise during checks:

• Network issues on the server side. Virtually unpredictable. Backup does not fly to the cloud.

• Overloading the server in terms of resources (low disk response speed, low server performance) and, as a result, freezing of the copy creation script. Forecasting is not amenable, you can only accept and check.

• The free space on the client server has run out (the size of the site has grown, the cache of any component has swollen and needs to be cleaned, the logs of client scripts or the server have swollen and started to take up a lot of space, etc.). As a result, there is not enough space to create a copy and everything crashes. Solved by server monitoring and alerts to admins. We implement this based on Prometheus and Grafana.

• Out of cloud storage space. As a rule, this is a consequence of the previous paragraph. We also monitor in Prometheus.

Instead of a conclusion

Everything that is described above is suitable for Internet projects, including various kinds of CRM systems, self-writing and other “bitrixes” standing on a server with a data volume of up to 500-700 GB. Anything closer to a terabyte is not reserved just like that, and other approaches are already needed there.