How to easily "fly" into Hack and drag it into flow mode

Practice & Scale AI: Rewriting text at a level that allows it to pass the filtering of recognition systems), which we very “dashingly” flew into with the whole team (Anomaly Detection) and, looking ahead, I will say that we “dragged” to third place in our case. It is a pity, however, that there was only one prize place, but still it was a great experience and while the trail is still hot I will describe everything now and tell you what we did and how we “came” to our decision.

So, the hackathon took place on the platform PlantIT and the organizer was the company “Analytics Insight” The hackathon itself had several cases as its goal, one of which was the development of solutions for rewriting texts and, more importantly, generating texts and passing them (texts) through AI detection systems for generated texts (verification systems). That is, in fact, we needed to do the following:

1/ Get the source text and check it for generation, evaluating it on a 100-point scale;

2/ Make a rewrite of the text (generate) for SEO, suitable for human reading;

3/ “Camouflage” the text so that it looks as much like “human” as possible and passes through AI systems that catch generated content.

What was the business essence of the task that we were given (in my opinion): All existing popular media platforms actively fight against generated content, since, in general, clients are not interested in reading it, since it does not look “live” and is very boring, moreover, it feels like it is compiled according to some kind of template. Hence, the departure of clients and the “failure” of advertising companies on the media platform, and as a consequence, the loss of invested investments of advertisers. There may also be other applications of this solution, but this approach seemed to me the most relevant, at least, I have been observing it for a long time.

Preparing for the hack and command

Based on the tasks set, we actively set about solving the problem, a huge amount of material was reviewed (texts, articles, models, analytics, blogs, chats – in short, where have we not been ;)) ). We went through a huge number of models, something “flew out of the box”, something had to be tuned and finished “hot”. At the very beginning, during a general call, we decided what we would do and in which direction to “dig”. I (the team captain) distributed roles in the team taking into account the wishes of the guys and their experience (yes, our team is called “Anomaly Detection” we constantly participate in hackathons, and this is not the first).

So, I (Vladimir Mironov — DS, Analytics) was directly involved in selecting models, testing and tuning them, analyzing and developing ideas for “camouflaging” texts, as well as general project management from the business requirements side and communicating with the customer at pitchings. Zhenya (Evgeny Gordeev — DS, Analytics) was given the role of analyzing and assembling everything that had been developed for the models, as well as tuning the models, launching the service, its support and critically analyzing the tasks set. The organizers immediately indicated that the main thing was to assemble the core of the project, without “bothering” with the design and “cosmetic” aspects of the API. Igor Shumsky (Analytics, Visualization, Support), as a beginner, but already participating in hackathons specialist, monitored the entire process and was engaged in developing a presentation reflecting the key aspects of our solution, all its subtleties, simultaneously delving into how the process is going and gaining experience. Vanya (Ivan Ushakov – Support), carried out information support for the project and constantly searched for information from various sources, throwing it into the general chat. Roma (Roman Solonchenko – Support, DataBase), monitored the entire process, and also suggested interesting ideas, and in the event of an unforeseen situation or absence of someone, he could quickly join the development.

The hackathon lasted three days online. The work was fun and energetic, the team worked harmoniously and calmly, all issues that arose were resolved in the working order very quickly. They worked closely, but without hysterics and fanaticism, if the guys needed to leave on their own issues, this did not cause problems, for the rest of the team, everyone was in touch, no one bothered anyone, it was great that we managed to build a calm flow-mode. We “dashed” from morning until late evening, but at night we slept and everyone rested, since the team needs rest and accumulation of resources to generate ideas.

Before each checkpoint and pitching, we had short calls to discuss what we would say, how best to present everything, and at what stage everything was implemented. For the final pitching, we prepared a large presentation and recorded how the application actually works.

Project and service

During the technical implementation of the project (in such a short time), as I already said, we went through a bunch of models, searched everywhere, on githab, huggin face and just ran around the network. At the same time, as a model that will analyze that the text was generated, we took Juner/AI‑generated‑text‑detection Today, this is one of the top models for solving this class of problems, it was tuned and configured. Here I will say right away that we have implemented a project that works within the company's perimeter, it does not transfer data outside, and does not require connection to an external network, which is an undoubted advantage when implementing the task.

The paraphrasing of the text was carried out using the following class of models: T5, Bart, T5+Parrot, Opus, PegasusPegasus + PAWS + T5. During development, we used a standard library pack, there were no frills here: pathlib, re, argparse, logging, itertools, concurrent, tqdm, torch, transformers, sentencepiece, nltk, Streamlit. The model was evaluated using the BLEU metric. In addition, we took into account the option of presenting the solution (rewriting) not only in English, but also in Russian, and presenting not one rewriting option, but several, so that the user could independently choose the option he liked.

While working on the task, we “ran” a large number of tests, some were successful, some were frankly “failures”, but we found out the main subtle points of rewriting, as applied to the task of “camouflaging” the text. We used both pure models and mixed ones. In the following figures I presented the test options.

Fig. 1. Rewriting obtained by means of T5 (Here we did not obtain any special quality, the BLEU metric was at the level of 0.06, which is very low, and as you can see the quality of the rewriting suffers due to the loss of a large amount of information)

Fig. 2. Rewriting obtained using BART (Here we got a clear breakthrough at the level of 0.67, which turned out to be very pleasant and gave food for further development, and the volume of the text and the general meaning did not suffer much);

Fig. 3. Rewriting obtained using T5+Parrot (Here everything went “below the baseboard” in terms of metrics, so we abandoned this architecture from the start; the Pegasus + Paws + T5 combination outlined a similar situation for us, but we left it for some “specific” texts);

The most breakthrough solution turned out to be a very inconspicuous network (the organizers at the pitching, it seemed to me, even lost the power of speech for a couple of seconds, and said that the solution was extremely unusual, but quite functional) Helsinki-NLP/opus-mt-en-ru. That is, we used the double translation model, which is especially good, the network does not require significant resources, and the quality of rewriting is very high (Fig. 4.).

Fig. 4. Rewriting obtained using Helsinki-NLP/opus-mt-en-ru

If at this stage we got a good solution, the next important point was “camouflage” of the text, a kind of post-processing of the text. A very interesting task, in my opinion, and not the most trivial. Here, a comprehensive analysis of human errors and writing features when writing a text came to the rescue, such as:

punctuation errors and double spaces (Fig. 5);
omissions of letters and incorrect stylistic construction of sentences;
extra punctuation marks and changes in sentence length (Fig. 6);
rearranging words in a sentence and replacing passive constructions with active ones (Fig. 7);
adding emotional elements and replacing words with colloquial equivalents;
various writing styles (academic, professional, technical);
replacing words with synonyms (Fig. 8) and much more.

Of course, there are quite a lot of these approaches, and a well-designed SEO text corrects many of them, but anything can happen, and the editing algorithm does not always see everything, but that was the “calculation” that the model would even more strongly consider that the text was written by a human.

*Fig.5. Introduction of double spaces and extra punctuation marks*

*Fig. 6. Rearrangement of words in a sentence and change in the length of sentences*

*Fig. 7. Replacing words with colloquial analogues.*

As a result, we managed to implement a very simple, fast and effective template-algorithm for rewriting texts in the broadest sense, that is, we tried not to be tied to a specific industry. At the same time, before the start of the competition, we “combed through the site” of the customer's company and realized that our customer's clients are the most diverse and it is very difficult to take into account all the specifics of the activity. Therefore, we decided to write a more or less unified algorithm and approach to solving this problem.

All the results obtained are shown in Fig. 8. It shows the work of the algorithm, which finds out whether the source text was generated and gives an assessment of the degree of generation. At the same time, along with the top solutions, we still added not the most relevant models, since depending on the style of the text and its structure, they can also give a good result, but their range is not very wide.

*Fig. 8. Evaluation of the degree of generation of the text supplied to the input*

Thus, we took 3rd place with not the worst score, in total about 11 teams reached the final. At the general pitching, very interesting approaches and ideas from other developers were also presented and their solutions were shown. At this point, the hack ended and 4 hours later, after checking all the solutions, the rating and the result were announced with the “rollout” of the leaderboard (Fig. 10). Special thanks to the organizers for the high-quality preparation and holding of the hackathon.

Fig. 10. Certificate (all team members received such a certificate with their name)

So fun and simple we dragged this hack in, by the way, I have blogwhere I publish the latest development news on ML & DL and data analysis, as well as a little analytics. There are short excerpts from articles and notes from other developers' blogs. There is also chatwhere you can ask any question and I will be happy to answer it, there is also a group in VK and a small channel on YouTube. Channels on other platforms will appear soon. Subscribe, ask questions, I will be glad to see everyone.