Why am I disgusted by hackathons, but I still participate in them

Hello! This article will be about our “expert” look at hackathons, where we will throw in a couple of holiriv take-downs and, by the way, talk about our solution for True Tech Hack.

Living in Innopolis is incredibly boring. So much so that in our IT village, in three years, only one more technopark was built, which was not needed by anyone in the post-COVID period of remote work. So we decided to go to unwind. And the choice fell on the True Tech Hack hackathon, timed to coincide with the recent rebranding of MTC. Of course, in order to attend the final stage of the hackathon (also coinciding in dates with the conference of the same name, which is definitely delicious), you need to get into the top ten finalists, but more on that later.

This year two problems were presented by the orgs and both of them are not bad in fact. Here’s a short summary for you:

1. Adaptation of films for people with special needs (some variability of filter settings and display of content in the video player is assumed). Solutions are considered in the form of an additional menu to the existing Kion functionality;

2. Audio accompaniment of what is happening on the screen for people with visual impairments (in audio commentary format). Solutions are considered both with pre-processing and with real-time video stream processing; An important criterion is the cross-platform nature of the proposed solution;

Of course we took the second one! See for yourself, here you can try to sit on two chairs, supporting both streaming and preprocessing of video content, and look at the bottleneck of the cross-platform in a real case, not to mention the cumbersome, but extremely entertaining data processing task. And all this in one request for a week, beautiful …

The choice of the format by the MTS team was extremely justified – as many as 5 days with several checkpoints for the implementation and testing of ideas for such an extensive task fits perfectly. The guys also provided all teams with promotional codes for access to the MTS Cloud and prepared repositories for working on solutions in the Gitlab. For this, the org team definitely likes it.

Here we have clearly indicated how much good there is in the initial commit of the hackathon flow. We shook our hands, looked and thought – the teams will start writing solutions and the problem is no more. But it’s not. Something is bound to go wrong.

Let’s stop here, take a look around and talk about hackathons as a concept in general.

We have seen many times how mentors show no interest in technical pitching if the solution stack does not match with their main one. In their head, something like “I don’t understand this, why should I go into details” probably sounds. When you see this, you immediately think – well, everything is on point, but then why mentor teams at the hackathon? This situation quickly turns into the problem that middlemen have nothing to do at the hackathon. They absolutely do not grow in skills, working in the format to quickly roll out a lot of features in a couple of days. After all, you do not call versatile-competent developers into experts. People who want to chill and chat on a business trip for the company’s money most often get there.

The situation is even worse when the team of experts and / or mentors consists entirely of PMs, products, sales, leads and other managers who have little to do with the technical part of the project. Then this is not a hackathon at all. This is an illusion that we are playing with because we don’t really want to make any decision. We want to play startups. But I am glad that they are all kind and positive. After all, soft-skills decide here.

At one of the hackathons, there was a designer in our team, he thought for a long time what color the buttons should be and in what format the cards should be shown to the user. As a result, the global impact from him was not too much. But his opinions about each view were being developed for a long time. Each person involved in the solution increased the queue and the execution time of the planned features. After all, it is almost impossible to create a distributed flow of work. There are a lot of practices to deal with this – we split all tasks into sprints, formulate deadlines, get special people to monitor the turnover of tasks. And in the long run, going to the prod works well, but not at the hackathon. There are no fats and scrum masters, no time to wait for new edits or an updated version of the design.

We make moodboards in the process of developing solutions. And it is much more productive for brainstorming and/or creating a full-fledged design.

It’s easy to work like this. You can spend time and write something non-obvious, feel the limits of your favorite ui-framework. By the precepts of the community. Or quit and not spend money on ridiculous custom controls. Transfer forces to something more significant to solve the problem, and take all implementations of views from the popular cupertino / material lib.

Many teams think that the only right task at a hackathon is to solve a business case. And they are right, but in practice we all hate it. But for us it is manifested in everything. First you give up on styleguides and linters, and the code ceases to be pleasant for all team members. At the design stage, you ignore the importance of API convenience, and write code, considering it as a whole draft. And then you merge everything into the master, without having time to return and fix the already formed technical debt so that the code remains readable and scalable. The problem is that your small project for a few days quickly grows into a huge unsupported codebase of dubious quality closed to modifications. And it doesn’t get fixed. Because most regular hackathon participants are not used to writing understandable and flexible code right away. They write code that can only solve a business problem.

The worst thing is that solutions do not win because of quality. Managers took over the hackathons and made soft skills more important and significant than hard skills. Presenting a solution is obviously a job for an extrovert. Beautiful slides should include jokes and convincingly communicate the importance of an issue that has already been raised by the business, even if it has been discussed multiple times. And in the team there should always be a sociable person who can cover this role. After all, how your team will show itself is not determined by how many HRs will approach you to take contacts. Namely, how many points your decision will receive as a result from experts.

If you have read up to this point, you either agree with us or are going to the hackathon, and soon you will feel all the takes in your practice. But the world is not black and white, and even in the dunghill sometimes come across pearls. A hackathon is an opportunity for a green jun to make connections, and a bearded senior to try a new stack. And knowing all the pitfalls that we tried to highlight above, you will most likely be able to win, just as fluctuatingly as wolves successfully pass technical social security.

You don’t have to look far for examples – just look at our solution for audio commentary in Kion. How we came up with the idea and design you have already read above. So now we can talk about how our implementation came out.

We have developed a system that complements the silence while watching a movie with comments from the model through multi-modal frame analysis. This allows you to get a similar experience from watching a movie to people with and without visual impairments. And the created system uses various neural networks tailored to solve specific problems. Thus, it remains easily expandable and reasonably accurate.

Orgami was asked to take as a basis any available open source media player. Therefore, we decided to take the popular plugin from flutter team from pubdev.

The UI/UX itself is as close as possible to the original Kion. And for the TV version, we even bought a TV set with a certain operating system and started there with the same codebase as for the mobile and web versions. It’s cool!

Let’s talk about how it works step by step:

What’s going on under the hood? Processing one frame of video creates a delay of 40 + – 2 ms, which is why it was decided to process video fragments in advance, as well as cache the text format of subtitles for duplicating voice acting. When saving a fragment, two audio tracks are superimposed with an amix filter, which allows you not to artificially mute any of the tracks. Frames are processed once every fixed number of seconds (default = 3), other frames are skipped. The dubbing of the scenes is designed for specific moments in time and the replicas are inserted intermittently if the time of the desired frame has not come up.

• If we see a person, we try to recognize the character, as well as his emotion. (a model for recognizing faces and matching them with the faces of previously known characters and a model for determining the emotions of heroes)

• We use a text model that connects facts into a concise description.

Here, generative transformers are used to transform the image2text type (a description of what is happening on the stage), the models are taken with a hugging face

Here, a generative transformer is used to transform the text2text type (paraphrasing what is happening on the stage, taking into account the characters, their emotions, age). Connected api ChatGPT 3, without translation from English.

The service works in streaming mode, the delay is 42–76ms (when using locally raised text models), but it appears only at the beginning or with a weak Internet, later the video is buffered, the experience is like on youtube. Pre-processing is also possible, while there is no need to change the source file of the film, we have written a custom player in which it is possible to use a separate audio stream.

Our proposed solution has many disadvantages in terms of the amount of resources required. And just closing your eyes to them will not work.

• Disk space: The weights for the neural network require space, as do the generated subtitles and voice recording. However, compared to the weight of the films themselves, this is a relatively small amount.

• Computing power: processing a movie with a neural network requires significant computing resources, however, they are required only during the first streaming viewing of the movie or during pre-processing. In the future, thanks to buffering and saving the result of the work, the load will be at the same level as when watching a movie without accompaniment.

• Bandwidth: for a streaming service, the availability of the broadcast for all users is very important. On-the-fly processing implies that the algorithm will also “watch” the broadcast, this is necessary once for each fragment of the film, so the peak load will be if N open different time periods in a new (raw) film. Then the load on the network will be 2N. However, let’s be objective, the viewer, most often, watches the film consistently and, especially in relation to the new film, such a situation is unlikely.

• Training data: To improve recognition accuracy, a significant set of test data is needed. This is optional, at the request of the site.

• The work of moderators: the result of the work of artificial intelligence is not guaranteed to be as expected. Despite the fact that you can fine-tune the result very finely and competently, moderation is recommended.

• Cost: The resources listed above need to be purchased or maintained, so it may make sense to provide an audio accompaniment service for a specific set of films.

We all love to complain that things don’t work out the way we expect. But in fact, this “not so” already contains many positive things. And so our team rolled out exactly such a solution as we saw it at the moment of those five days during the hackathon. Now, of course, we see in it a crazy lot of problems and are actively refactoring and, in fact, redoing everything. But such are hackathons, and we love them for this spontaneity and speed of development. C’est la vie

Separately, we want to express our congratulations to the rest of the hackathon participants and thank you for the opportunity to listen to other solutions, as well as look at the problem from a different angle! Thanks also to the guys from MTS for an interesting event and a good organization;