The scheme for counterfeiting, restoring and checking fruits in a greengrocery. Excerpt from a book
Hi, habrozhiteli! So far, such a situation outside the window, we decided to share an interesting excerpt from our book Generative Deep Learning. The creative potential of neural networks ”by David Foster.
Apples and oranges
Granny Smith and Florida are co-owners of a grocery store. To achieve maximum efficiency, each takes care of its shelves in the store. For example, Granny Smith is proud of her choice of apples, and Florida lays oranges for hours, achieving their ideal location. Both are so confident in themselves that they agreed to a deal: profits from the sale of apples will go to Granny Smith, profits from the sale of oranges will go to Florida.
Unfortunately, neither Granny Smith nor Florida intends to compete honestly. When Florida does not see, Granny Smith makes her way to the window with oranges and begins to paint them red so that they look like apples! Florida does exactly the same: she tries to make Granny Smith’s apples look more like oranges, using a spray of a suitable color when she turns her back. Because of this, customers often choose the wrong fruits they intended to buy. Those who were about to buy oranges sometimes mistakenly choose Florida colored apples, and those who wanted to buy apples mistakenly choose oranges masked by Granny Smith. In the end, the profit for each fruit is added up and divided according to the agreements – Granny Smith loses money whenever one of her apples is sold as an orange, and Florida when one of her oranges is sold as an apple.
After the store closes, both begin to restore order on the fruit shelves. But instead of trying to wash the paint caused by the rival, both simply paint the fakes, trying to bring them back to their original form. It is important for them to do this job correctly: if the fruits do not look as they should, then they will not be able to sell them the next day and again lose profits. To make sure their actions are correct, they sometimes test their methods on their own fruits. Florida sprinkles its oranges and checks to see if they look exactly the same as the original. Granny Smith tests her “skills” on apples. If they find obvious discrepancies, then they will have to spend hard-earned profit on the study of more advanced methods (Fig. 5.2).
At first, buyers often make the wrong choice. However, over time, they gain experience and learn to determine which fruits were faked. This forces Granny Smith and Florida to perfect themselves in faking each other’s fruits, as well as in restoring the fakes to their original appearance. In addition, they need to periodically make sure that the techniques they use do not alter the appearance of their own fruits.
After many days and weeks of this ridiculous game, they noticed that something amazing had happened. Customers no longer distinguish between real and fake fruits. In fig. 5.3 shows what the fruits look like after falsification and restoration, as well as after testing.
CycleGAN
The previous story is an allegory describing the development of one of the key areas in generative modeling, known as style transfer: a cyclically coordinated generative-competitive network, or CycleGAN. The article describing this network is a significant step forward in the field of style transfer and shows how to train a model that can copy a style from a set of reference images to another image without using a training set of paired examples.
Pre-existing style transfer models, such as pix2pix, required that each image in the training set exist in both source and target form. For some style transfer tasks (for example, for converting black-and-white photographs into color photographs, maps into satellite imagery), you can create such a data set, but for others it is simply not possible. For example, we do not have original photographs of the pond from which Monet wrote his series “Water Lilies”, and paintings by Picasso with the image of Empire State Building. By analogy, it would take tremendous efforts to get photographs of horses and zebras standing in identical poses. The article describing CycleGAN came out just a few months after the article describing the pix2pix method and shows how to train the model to solve the same problems, but in the absence of pairs of source and target images. In fig. Figure 5.4 shows the difference between paired and unpaired pix2pix and CycleGAN datasets, respectively. If the pix2pix method allows you to model changes in only one direction (from the source style to the target), then CycleGAN trains the model in both directions at the same time, so that the model learns to convert the source images to the target and vice versa. This property is due to the architecture of the model, so you get the ability to convert in the opposite direction automatically.
Let’s see how to build a CycleGAN model using Keras. To begin with, let’s take the previous example with apples and oranges as a basis, go through all parts of CycleGAN and experiment with its architecture. Then we apply the same approach to create a model that can apply the style of a particular artist to the photograph of your choice.
Your first CycleGAN network
Most of the following code was borrowed from the Keras-GAN repository (http://bit.ly/2Za68J2), maintained by Erik Linder-Norén. This wonderful resource contains many examples of generative-competitive networks from various books implemented with Keras.
First you need to download the data that will be used for training. In the folder where you cloned the repository with examples for the book, run the command:
bash ./scripts/download_cyclegan_data.sh apple2orange
In the downloaded set of images of apples and oranges, which we will use later, the data is divided into four folders: trainA and testA contain images of apples, and trainB and testB contain images of oranges. That is, A represents the space of images of apples, and B represents the space of images of oranges. Our goal is to train the model on the data sets in the train folders to convert images from set A to set B, and vice versa. Images in the test folders we use to test the quality of the model.
»More details on the book can be found at publishing site
” Excerpt
