Automation or how I avoided communication with colleagues. Part 1
I worked in several high-tech companies and teams, and not all of my colleagues had the desire to at least try to transfer something from routine to automatic or semi-automatic mode.
Case #1 – My very old example from 2021
I tried to automate the process of uploading and transmitting reports.
Don't get me wrong, I love people, and my colleagues too (yes, they are people too). But when they urgently need something right now (ad-hoc), and there are 30 people in need, then I love them a little less.
Problem: limited time and work resources
At one of my previous jobs, where there were only 2 analysts in my department (AdTech), and the need for reports grew with the number of launched advertising (marketing) companies and attracted clients. Every month (and/or week) take out and put in everyone pictures, reports, metrics and predictions for 100 years forward and backward (or at least a month) and more. There were many companies for which we did this, so many, different sizes and categories. And like clockwork on Monday, on the Nth day, everyone came to us in the morning and, like seagulls, “give-give” a report (yes, this is also part of their work).
I thought and thought about how to relieve this ill-fated Monday so that the wolves would be fed and the sheep would be safe. After all, no one has canceled other tasks; they can be both important and urgent.
First, let’s understand what the process is and what its stages are:
What, when, for whom the marketing campaign was carried out – this is the data that goes into the report. Metrics are calculated based on them and graphs are built. This is especially important if the company as an organization (client) is one, but it launched several different advertisements, at different periods or for different durations (the type of advertising also differs: some are for brand recognition, some are for conversion, etc.).
Collecting data and storing everything in a database.
The report should appear on the desk of each account manager on Monday and preferably in the morning for each conducted (launched) advertisement separately. (hereinafter the company will mean the advertising company, and the client will mean the company as an organization)
The account manager “checks” the report and sends this report to his clients
This whole process took 2-3 hours at best, and 24 hours at worst.
I started from the end, because I didn’t want people to start writing to me at 8 in the morning, and at 11 they were already swearing that the reports were not ready yet, we needed to somehow prepare them in advance, or limit the communication itself without sacrificing productivity.
Attempt No. 1
For example, I tried to prepare reports to the Armed Forces the evening before.
But there are 2 disadvantages here: working in the Armed Forces, and the fact that the client wants to see the full reporting period (what if someone buys a TV for a million rubles at 23-30 Sun, and we will not display this in the report), and even a delta in 1 day not acceptable.
Attempt No. 2
Then, you need to set up communication within the team, so that everything works without kicking. Well, for starters, I decided to organize a shared storage for us on (I think) SharePoint (in general, the mechanics of working with cloud storage are +/- the same everywhere).
There, separate folders were created for each company – within folders for years and months – hierarchical directories.
Now I would..
Now, in this jungle of folders, I would add another “archive” folder, where I would send all outdated reports with a specified frequency.
I distributed access to employees (a specific account manager can have access to a company folder, or just everyone), access to the folder with the report was given to external clients in a strictly regulated manner.
Now I would..
Now, I would also add a script in any language that would be launched by the orchestrator (Dag on airflow or even an internal scheduler program on a work machine – it also runs automatically according to a schedule) – to send a sample by email to the account manager and company representative.
Even after this step, it became much easier when colleagues got used to the fact that everything is now stored and appears in one place. But this did not increase the speed of reports appearing. And if my colleagues didn’t write to me at 8 in the morning, then they wrote at 12 with the question “when will it be, soon?”
And here, what exactly was the problem: the database was not created and maintained by me, and the scripts that collected the data were not written by me, this is our legacy code, I tried to give him artificial respiration and revive him. In addition, the report format was regulated, so you can’t go wild.
Actual problems:
The way the report was compiled required personal control (insert the name of the company, check that it still cooperates with us, indicate the period, and much more).
Speed of accessing the database and retrieving data.
And these were reports where data was only collected and aggregated – no ML models, or sitting and thinking about how to calculate benchmarks. Only melee.
I didn’t dare touch the database itself, and no one would have allowed me, so all that remained was to work with our internal script.
Attempt No. 3
I scratched my head and suggested this: before starting the script, let us always update the list of current active clients, their completed companies, and run a reporting collection script for each, formalize the variable part as much as possible, write down the rules (practically if else).
That is, I wanted to reduce our interference in the work of an already running program to a minimum. They did what they could. It became even better, but not good enough, unfortunately, for correct reporting it was necessary to work with a large number of variables, which. Not at all or not at all amenable to formalization. So there were exceptions to our rules, and therefore mistakes. But there was only one script (template), and with the first error everything crashed and reports stopped being generated.
While we are in multitasking mode, we notice it (the error), some time will pass, which means that we will again receive a dozen “and when is the report?”
Attempt No. 4
“Let’s ignore the mistakes,” I suggested cheerfully, and my colleague supported me.
As a result, we somehow found a plugin for Jupiter laptop that did not allow the script to stop running, even if it encountered an error – after all, we have separate independent blocks. And then we looked at which reports were uploaded and which were not.
We created separate scripts for functionally independent elements, which were called according to conditions in the main script, for more transparent debugging and the formation of a ruled-based approach.
Now I would..
Now, I would create an additional text file – where I would simply print a list of downloaded reports and where the error occurred. You can put it on a ball or send it by email to analysts
At that moment, I was not familiar with airflow, and my colleague did not tell me anything. And we set up the launch automation somehow crookedly, or maybe we didn’t set it up, I don’t even remember.
Yes, and although this cannot be called complete automation, the efficiency in terms of saved time and nerves was decent (on our part, and on the part of account managers, and on the part of clients).
My general conclusions and personal life hacks:
If something is repeated more than 3 times and tends to continue (even with changes or adjustments), then it’s time to think about some kind of helper crutches.
If something took more than 2 hours and is not purely intellectual and creative work, you can also think about it. From practice, if something happens once, it will most likely happen a second time.
I love information; many people, of course, are annoyed by the streams of automatically sent letters – they help me track the process and collect feedback.
As a bit of self-irony, I’ll attach this picture.
In the next post I’ll tell you, with code examples (I hope):
How we did auto-check of homework.
They forced managers to count numbers and write them down on a table.
Maybe I'll remember something else
