How did we automate FTP(S) with unique problem symptoms?

My name is Borat. I love you. I love Aichi. Nraitsa!

Borate

Loves FTP automation

This story will tell you how our company automated file transfers over FTPS in conditions when standard solutions are not suitable and developers have thoughts to fork ftplibwhich is actually more like being depressed about this task.

The company I work for is in the business of DaaS.

At one of the stages of our pipeline, we faced the problem of automating the transfer of files to the client via FTP.

We couldn’t solve the problem, so the idea came up to automate button presses for FileZilla.

We used: wmctrl And xte.

• wmctrl – Helped manage the window FileZilla;

• xte – pressed buttons and hotkeys.

This is a macro designed for GUI – local rapist Not good!

Borate

Outraged by GUI macros

GUI automation is a strange solution

Because it did not guarantee the delivery of files to the client. Working with GUI automation, we cannot accurately trace errors, check parameters, conditions, see the details of any part. Apart from logging, of course. FileZilla provides us with such an opportunity, but this is “proof against the contrary”. It’s much more important to start by catching errors in the first step – connecting to the server. So that you can at least reconnect to the FTP server.

So. The client server uses FTPSyou need to explicitly specify the port, IP, authorization data.

FileZilla has the ability to save login options, and we can even use a command line option to start the login client by specifying the name of the saved data from the session manager.

But this also does not make sense, since it is impossible to track in which directory FileZilla is currently located..

In the course of using the client, we found that if there are only two folders in the local directory: [‘..’, ‘some-dir’]then, when selecting through CTRL + A and pressing Enter, we will be transferred to the directory above, since FileZilla has such a bug and clicks on the “..” folder, which is a transition to a level up (thanks, cap).

Task: Send files to the client server via FTPS. Provided that the client’s router as a response when connecting via FTPS gives internal IP.

Despite the many Python modules, we had an extraordinary situation that could hardly have been foreseen by the developers of such well-known libraries as pyftp, sftpy, ftplib.

In companies strictly regulated by documents (our customer is such) – the client’s networkers sometimes do not have the right to change something for the contractor, so on this way we met SUCH bad option. And it had to be solved somehow.

About FTP packages for Python

All ftp clients for Python just hung when connected. There is no answer, no hello.

Hundreds of articles have been read, tons of documentation and a huge steak video has been reviewed. As a result, we later realized that the server returns us a completely inappropriate IP address after connecting.

All these libraries take this IP and use it in further operations. For example, when authorizing, a request is sent there. Why is this done? We already manually set the IP we need. These are the jokes we received from the developers of these modules.

Path to Solution

By the way, the standard FTP in the Linux terminal did not suit us either. He didn’t know how to do FTPS transfers.

It turned out that when querying Google, such as “cli ftp clients” is often mentioned lftp and others.

Alexander V. Lukyanov

Developed LFTP in 1996

So the day fell that we still tried LFTP, and the answer was already different! At least the request does not hang up. But something doesn’t show up either.

And the problem was that the SSL certificate did not pass verification. It’s probably just self-signed. Luckily, this can be turned off. Because LFTP has powerful settings. Here: and logs, and the path to the logs, and SSL settings, and much, much more.

Code – examples

LFTP can transfer a bunch of files to the server and log the work process:

lftp -u username,password ip << SCRIPT

set xfer:log-file "/path/to/your/file.log"

cd /home/xxx/dir/ # перейти в директорию удалённого сервера

mput *.logs # множественная выборка и отправка подходящих файлов

SCRIPT

LFTP can mirror two directories, that is, transfer files to the server, but not overwrite existing ones, and even in 10 threads:

lftp -u username,password ip << SCRIPT

mirror -R -P 10 /src/dir /dest/dir/

SCRIPT

Nraitsa!

Borate

Automated FTPS

Author of solution, code and article: Denis Kovtunenko