IPhoto library’s path to cloudy Yandex.Disk through stars and thorns

Somehow I came across a successful distribution Yandex cloud storage and got hold of it 1 TB. The first thing that arose in my head was to put my iPhoto Media Library She lives on my external hard and the volume is neither more nor less – 800 GB! It would seem that something complicated – took yes uploaded to the cloud, but …

Let’s take a quick look at how architecture works. Yandex.Disk and similar decisions. Actually, everything is ingenious and simple: there is a folder in the cloud, there is its analogue on the local disk.

Everything that fell into the cloud is pulled to the local disk and vice versa. Brilliant and simple. However, it turns out to pour their 800 GB you must have at least one disk space in your unhappy cloud from an external hard drive SSD – on a working laptop! In general, the path of grandmothers using a regular Yandex.Disk client does not work here.

We go to read the service manuals and find out what they have Webdav– Hallelujah, here’s the solution! We mount the volume into the system and off we go. But we understand that the volume is large, it will pour more than one day, it will have to be interrupted for one reason or another, and resume filling.

Good old man comes to the rescue here rsync – The final version of its use is given below, but for now let’s return to a not very inspiring moment in this whole story: pouring speed. Alas, it is not high – it contributes to the limited channel, of course, but the main thing is sagging on remote file operations (mainly, fsync of course) that must be executed for each copied file.

Accordingly, when copying gigabyte videos, this against the general background is not very annoying, but when there are a lot of fairly small files (photos, in fact), it saddens unusually.

Let’s go read the protocol Webdav and we see all kinds of things there ala MPUT – we’re wondering if there is any magical client who effectively does this and according to reviews under Mac OS very praise Cyberduck.

In appearance, it really magically begins to fill up quickly – it’s as if on a local FS everything is happening. A more detailed study showed that everything really happens – it doesn’t work at all somehow super optimally with Webdavusing MPUT there for small files, but exactly like Yandex.Disk – i.e. it has its own local cache (it was experimentally established that it is limited approximately 1 GB) and, accordingly, it creates an imitation of fast file operations locally – it responds as an FS driver to the application, but in fact it simply writes them to its cache and quietly syncs through Webdav.

Accordingly, let on it rsync, which first copies the file to a temporary one in the target directory, and then does it move– no sense whatsoever – Cyberduck just wedges on this topic – for now it reaches the file in its background synchronization – that is already gone.

Nothing to do, I poured through the classic client Mac OS slowly but surely order 10 days. It’s clear that I had this business for 10 days interrupt for one reason or another. The Internet used to disappear, but given the principle of operation rsync, I did all the refills using the option –ignore-existing – if rsync I successfully uploaded and renamed the file + given that I did not change the source files at that time – I didn’t use iPhoto, the fact of the presence of the file on the target medium is quite enough, there is no need to check the checksums and synchronize the changed files.

Subsequent topping up of differential backups should be done using the option –size-only – photos / videos do not change as a rule – only new ones are added or meta-information is changed – all kinds of media library database files, etc. – in general, the correspondence of the file size on the source and the target media is enough to conclude that its synchronization can be skipped.

Another point that had to be taken into account when creating a backup of the library: as I said at the very beginning, the library takes more 800 GBbut it’s not the worst thing – the most unpleasant thing is that the files there 373K! Moreover, as it turned out the photos / videos themselves (in fact, the most important part for me in terms of backup) is somewhere 70K, and everything else is useless media thrash – all sorts of faces there iPhoto looking, etc. – in general, the lion’s share of these files is located in directories resources / proxies and resources / media in the library directory and occupied a total of 33 GB in my case.

Given that this is essentially a backup and use this media library directly from iPhoto we won’t do it: we create 2 archives for these directories – at least with the help of tar without compression – there jpegwhich all the same does not really squeeze. Further using directives rsync –exclude skip these directories and all the contents in them. Those. The final team looked like this:

rsync -axv --numeric-ids --progress --ignore-existing --exclude="/resources/proxies/***" --exclude="/resources/media/***" /Volumes/TOSHIBA EXT/Медиатека Фото.photoslibrary/ /Volumes/webdav.yandex.ru/Медиатека Фото.photoslibrary/

Well, that’s probably all – we’ll carry out further refills, as I said, using the directive rsync –size-only. There is still trouble from Applethat change the directory structure or data storage format – but here, I think, the directive will help us –delete (prescribes rsync delete non-existing files / folders on the target resource during synchronization). Or we’ll see what they come up with there – where ours didn’t disappear.

P.S. I hope that my experience will be useful to you. There will be questions, write, I will be happy to answer. Thank you for the attention!