Pipeline for personal photos and videos
If you've ever wondered how to store your growing personal media files, you have an Android phone and you don't really trust the clouds, then this article might be for you.
My 948.4 GiB media storage looks like this now (photos start in 2001)
And the Pipeline itself is drawn on the diagram. Explanations are after the picture.
Appearance of files
In 97% of cases, files appear on the phone. Either directly when shooting with the phone's camera (photos in HEIF, videos in HEVC) or through import from various action cameras.
With the camera phone everything is simple, photos are viewed and unsuccessful ones are deleted, videos are cropped in the native Samsung editor or edited in CapCut
With imported from action cameras – it is more complicated. First they are imported to the phone, then edited, then the resulting is poured into a folder that participates in the general pipeline, and the source files are mercilessly removed. With GoPro and DJI Action 2 this process was not very urgent, due to the fact that the files were not very large, and the native editors are not very convenient. With Insta360 X4 this whole process is much more dynamic. Firstly, because the source files are simply huge (1 minute 8 seconds in 8K 30fps is 1.4Gb), and secondly, the application is very convenient, on Samsung S24 Ultra it works very quickly, both in editing and in importing into flat videos, and therefore this stage is not delayed and goes quickly and for fun.
There are also photos that are sent in messengers. The necessary ones are simply moved to a special folder for which synchronization is set up.
As a result, at some point in time there are photos and videos that you want to save somewhere else.
Google Photos
Google Photos is a dead-end branch in my pipeline. It is used mainly for instant saving of media files. As long as it can be paid for, it will remain there, if it fails to be paid for again or it is blocked, it will be easily replaced with syncthing as an instant backup. In 22, when there were problems with its extension, I cleaned it up, deleting all the videos and leaving a couple of photos from each event so that there would be a map and reminders of the events. In 23, the problem was solved, but there is no hope for it.
Synchronization in Nexcloud
Nexcloud is a server on Raspberry Pi 4 with an SSD drive connected via USB to Ubuntu and located at home. For greater security, it lives in a separate DMZ subnet that has access from the Internet. When all unnecessary photos and videos are deleted on the phone in the Google Photo program, and the necessary ones from messengers are transferred to the correct folder, then the jobka is manually launched on the phone in free version of foldersync. It is quite enough to sync new files to Nexcloud.
Nexcloud -> Immich
Server with Immich this is a machine on ASRock J5040-ITX in Thermaltake Suppressor F1 case with two 3Tb WD42PURZ drives in RIAD1 in 8G RAM. All this works on Ubuntu. Immich is raised via docker compose and since the project 'The project is under very active development' is updated manually via `docker compose pull`. Both the web ui and the native application notify that a new version has been released and there are no problems with this.
Every two hours, rsync is launched on this server, which goes to Nexcloud and picks up new files without deleting local files, even if they were deleted in Nexcloud.
There is also on the server Photoprism. I also went up through docker compose. In general, I started with it, it is good, but the web ui on the phone is terrible. I was looking for photos from 2005, and instead of closing the photo, I pressed back and flew to the beginning of the list, in 2024, then when scrolling, the web ui thought that you not only scrolled, but also selected part of the photos and you look, and you have not one photo selected but 15, and it is not clear where to look for them to remove from the selection, this could have happened unnoticed 10 screens ago. Immich has its own application almost devoid of these shortcomings. Since the bread prism does not ask for bread, but is updated automatically through watchtower then I left it, but switched it to read only mode, so that I could sometimes look at it and compare.
After rsync takes the files, Immich itself sees it, starts scanning them and adds them to the media library, and Photoprism does this on a kick according to a timer.
Immich works in RW mode with files, so you can delete files through it.
I can only remember one minus of Immich – it can't do raw. But I have a minimum of such photos.
Backups
Photos and video files that rsync drags to the server with Immich are backed up once a day using restic. Restic supports the same protocols as rcloneso it perfectly uploads backups both to Yandex disk and to home NAS via mounted samba share. All this is easily changed to any necessary and used in this part of the pipeline.
There are currently no backup tools for file markup in Immich in the pipeline, which is not very good, because information about albums and names of found faces is stored there. I have 709 faces in the recognized ones, and some names were really hard to remember through social networks and the phone book, it would be a pity to lose them. But this item is still in TODO.
Deleting files.
The entire pipeline is protected from deletion in several places, and therefore deleting files has some peculiarities.
It is best to delete photos during the initial cleaning in Google Photo, then they will go to the trash and you can get them for some time, and the space on Google storage will be cleared, which will not happen if you delete photos in the native gallery.
After the Nexcloud -> Immich stage, it doesn't matter what happens in Google Photos, on the phone or Nexcloud. The files will still remain on the server with Immich.
If you delete photos in Immich, they will still remain in the restics backups and to completely remove the file, you need to perform an operation to delete old snapshots.
Automation of deployment
The whole story described is quite easily automated in Ansible. That's exactly how I do it. There is only one manual operation – granting restic access to Yandex disk when deploying all this stuff.
