How Generative Models Work for Creating Text and Images

Hello everyone! My name is Anna Vasilyeva, Project Manager in the Category Management Department at Fix Price. I propose to consider in detail the basic principles of generative models for creating text and images: solutions based on generative models can significantly simplify the process of creating content and open up new opportunities for optimizing work processes. We will go over the key algorithms underlying these models, and also consider some application options for the selected solutions, which will allow us to better understand their potential.

Definition and main types of generative models

Generative models are a type of machine learning model that are trained on large data sets and are able to create data samples similar to those in the training set. In this way, generative models create new content, which makes them especially useful for tasks related to workflow optimization. There are several types of neural networks that generative models are based on, each with its own characteristics and areas of application.

Deep Neural Networks (DNN) — the basis of many modern generative models. They consist of layers of neurons, where each layer processes data and passes it on to the next level. They are used to discover complex dependencies and representations needed to create new data.
Autoencoders — are a special type of neural network designed to compress data. They first transform the input data into a simplified, or compressed, representation, and then restore it back to its original format. This process allows autoencoders to learn important features of the data and effectively reduce its size.
Generative Adversarial Networks (GAN) consist of a generator that creates new data by imitating real data, and a discriminator that tries to distinguish the generated data from the real one. The competition of networks gradually improves the quality of the generated data. GANs are used to develop realistic models for creating images and videos.
Variational Autoencoders (VAE) — an improved version of autoencoders that encodes data based on probabilities, allowing not only to restore the original data but also to generate new data. This makes VAEs especially useful for tasks that require generating new realistic data.

Now let's look at specific models that are used to generate text and images, and examples of their use in our company.

Text generation

The most famous family of models for text generation is GPT (Generative Pre-trained Transformer), which has been developed by OpenAI since 2017. These models are based on the transformer architecture, which has revolutionized natural language processing.

Here’s how it works. GPT uses an architecture consisting of self-attention layers and fully connected layers. The model is first trained on huge amounts of text data to understand the language structure, syntax, and semantics. This step is called pre-training. In the next step, the model can be further trained on smaller amounts of data to solve more specialized problems, which is called fine-tuning.

During text generation, GPT receives an initial sequence of words (prompt) and, using its ability to predict the next words, continues the text. The model uses probability distributions to select each next word, which allows it to create coherent and meaningful texts.

Where GPT is used:

Chatbots and virtual assistants. GPT is used to create conversational systems that can carry on conversations that seem natural.
Automatic writing of articles and contentThe model helps generate product descriptions, news and other types of texts.
Translation and summarization. GPT is used to improve the quality of automatic translation and to create short summaries of large texts.
Creative tasks. Generation of “blanks” for poems and stories, creation of scripts.

At Fix Price, we use GPT 4 within our video design studio to create and edit texts used in creatives, as well as to write and check the code needed to develop programs and utilities. One example of such use is the development of a piece of code for internal software that automates routine tasks and frees up resources for creatives and more complex tasks.

However, GPT is not the only family of models used to generate text content. There is also BERT (Bidirectional Encoder Representations from Transformers), a family of models that is also based on the transformer architecture, but with a key difference: it is trained using bidirectional context, meaning it takes into account both preceding and following words when training. This ability allows BERT to better understand the context and meaning of words in a sentence.

BERT uses a pre-training method that includes two main tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). In the MLM task, some words in a sentence are masked, and the model learns to predict them based on the context. In the NSP task, the model learns to understand the relationship between two sentences.

The main difference between BERT and GPT is the direction of text processing:

GPT predicts the next word based only on previous words.
BERT, on the other hand, uses a bidirectional approach, analyzing both previous and subsequent words simultaneously.

This makes BERT more suitable for text understanding tasks such as classification, question answering systems, while GPT outperforms BERT in generating coherent texts.

Where BERT is used:

Processing questions and answersThe model is used where it is necessary to find answers to given questions in large volumes of text.
Text classificationBERT is used for classification tasks such as text sentiment detection or document classification.
Data discoveryThe model helps to find and classify references to people, organizations, places and other information in the text.
Translation and summarizationWith its deep understanding of context, BERT is used to improve translation quality and create short summaries of texts.

As a business represented in several countries, we actively use artificial intelligence to translate texts from Russian into foreign languages. Fix Price follows this trend and successfully uses models to generate text in its work. Thus, we translate technical information about products using AI. Using neural networks, we also optimize the budget by reducing expenses on translation agencies.

Generating images

To generate graphic content, two types of models are used: DCGAN and StyleGAN.

Deep Convolutional Generative Adversarial Networks (DCGAN) — is an extension of traditional generative adversarial networks (GANs) designed to work with images. DCGANs use convolutional neural networks (CNNs) in both the generator and discriminator, allowing the models to efficiently process almost any image.

Where is DCGAN applied?:

Creating realistic images. DCGAN can generate images that look real.
Data augmentation. Generating new samples to increase the volume of training data in various machine learning tasks.
Style transfer. DCGAN can be used to transfer the style of one image to another.

StyleGAN (Style-Based Generator Architecture for Generative Adversarial Networks) — is an improved version of GAN designed to generate high-quality and photorealistic images. StyleGAN introduces several innovations that make it one of the most advanced models for image generation.

The main feature of StyleGAN is the separate processing of style information. This allows different aspects of the generated image to be controlled independently, such as pose, facial expression, color, and texture. StyleGAN also uses procedural methods to generate images, which allows for very detailed and realistic results. In addition, StyleGAN training starts with generating low-resolution images, and then the resolution is gradually increased. This improves the stability of training and the quality of the final images.

Where StyleGAN is applied:

Creating photorealistic imagesStyleGAN is used to create images that are virtually indistinguishable from real photographs of people, animals, and objects.
Editing imagesThe ability to fine-tune various aspects of an image allows you to create artistic effects and perform precise manipulations of images (for example, replacing certain fragments with more suitable ones).
Media content creation. Generation of avatars, characters for video games, animated films.

At Fix Price, we use tools based on the principles of generative adversarial neural networks to create visual elements and creatives for use in video production, advertising banners, and email newsletters. Designers use several tools to produce visual content. The first is Kandinsky. We use the model for storyboarding scenes. The neural network is easy to use and understands queries in Russian well.

The second generative model is Generative Fill in Photoshop. With its help, we draw backgrounds to images and create separate elements of static animation. We also use Generative Fill to create a generative background fill on video.

Example 1. Generating a background in Generative Fill

Example 2: Generating GIFs with Generative Fill

We hope this brief overview will help you understand how generative models that create text and graphic content work. As for specific products based on the technologies described above, this is a topic for separate articles. And perhaps we will talk about them in the future.