How to make interactive pixel images with D3.js?

Hola, Amigos! In touch Artem Saleev, technical director and Arseny Zakharov, frontend developer of a product development agency Amiga. Today we’ll tell you how we implemented a task for a large customer: to place “blurry” pictures on the website that would become blurred by user interaction.

For a large customer, whose name we are not disclosing due to an NDA, we have created interesting functionality at the front. The task sounded simple: you need to place blurry pictures on the site that will become clear when the user “erases” them with his finger (or cursor). In the implementation of this task, nuance after nuance was revealed, which I will discuss further.

You can touch what we got from the user side using this link: https://onsh-a.github.io/d3_puzzle/

For those who want to see the code, we leave a link to GitHub at the end of the article. And for those who are interested in how we chose this particular method of implementation, we will tell you about the course of our reasoning.

When the technical specifications were drawn up and the requirements became more or less clear, we went to the Internet to look for a reference. We were lucky and pretty quickly we came across such a demo. For the most part, this solution satisfied the customer's requirements, but not all.

Firstly, visually the picture did not look like a “pixelated” image of poor quality: in the demo the effect was achieved a little differently. There, the picture “appeared” from one element, from which it was impossible to guess what was depicted in the original picture. We needed to create an effect of poor quality in the original image, which the user could gradually improve through his actions – swiping across the screen with a finger or mouse, as if blurring it.

Secondly, the image and the number of original pixels were static and did not depend on the viewport. For us, the key condition was to make the mechanics in such a way that the picture on mobile devices always occupies almost the entire screen, but without having to try to hit small 2×2 pixel targets with your finger.

Thirdly, we had a requirement that we could calculate the speed of opening a picture and the percentage of remaining unopened elements. Fourth, the demo was written in closures, and debugging closures can be quite a difficult task.

Thus, the technical task boiled down to the fact that it was necessary to obtain a pixel representation of the picture, generalize it to larger elements and come up with functionality, using which the user could “improve” its quality. Large generalized factions would be broken down into smaller ones until only the original picture remained.

So the first thing we had to do was get the pixel representation data for the picture. To get the pixel data we use canvas. Using the method getImageData we get the pixel data of the entire canvas area. In property data there will be an array with a set of colors in RGBA format. If we remove all checks from the method, then obtaining color data will look like this:

public getColorMatrix(loadedImage: HTMLImageElement | null): void {
  const canvas = document.createElement('canvas');
  canvas.width = this.dim; // сторона квадрата картинки
  canvas.height = this.dim;
  const context = canvas.getContext('2d');
  context.drawImage(loadedImage, 0, 0, this.dim, this.dim);
  return context.getImageData(0, 0, this.dim, this.dim).data;
}

Next, you need to calculate what size the image will be. We also set a restriction that the images will be strictly square. For desktops, determining the size was easy. According to the design, the maximum image width should not be more than 512px. But for mobile devices, calculating the width was somewhat more difficult.

As we determined in advance, the minimum fraction should have been no more than 2px, and the maximum 32px. The value of the minimum fraction imposed the following restriction: the width cannot be an odd number. But that wasn't the only problem. The significance of the largest faction also brought problems.

Let's imagine that the screen width is 386px, in which you need to place a picture. 386px will fit 12 squares with a side of 32, but there will be smaller squares that we can’t combine into a thirteenth square with a side of 32 because it won’t fit on the screen, so they are combined into the largest possible parallelepiped.

In our case it will be 2px by 32px for the side edge and 32px by 2px for the bottom edge since the image is a square. Such a fraction will be barely noticeable to the eye and it will be very difficult for the user to hit it with his finger, even if he notices it. Therefore, the image width was calculated so that the minimum fraction was less than 6px in width or height.

Now we need to generate the markup. Here we laid down a logic that seemed simple and correct, but it still had to be changed – we will return to this at the end of the article.

We created an svg, inside of which there will be rect elements, and each element will represent a fraction of the image on a certain layer. Let's imagine that we are given a square picture with side n. It is divided into squares with a side of 2px. Next, based on the average color value of 4 neighboring squares, we calculate the color value of a larger square with a side of 4px and repeat this procedure, but with squares with a side of 4, forming squares with a side of 8px, and so on. (2 -> 4 -> 8 -> 16 -> 32).

For our mechanics, we chose a maximum square size of 32px. This point seemed ideal to us based on the fact that the image, divided into squares with a side of 32px, still vaguely resembles the original, at the same time, it’s easy enough to get into a square with such a side either with a mouse or a finger on touch devices.

Since such a solution involved a large number of manipulations with svg elements in the dom tree, we decided to use a library that is tailored to solve such problems – d3.js. It provides a large number of primitives for working and animating svg images out of the box.

Structurally, the application will consist of two classes:

As we wrote above, the original solution used closures. Although this solution worked, it was quite difficult to make changes to the code due to the fact that closures can be extremely unobtrusive. Once the image was initialized, it was very difficult to get its current state, which meant that the debugging process turned into a nightmare. It was necessary to come up with a structure that would be more convenient to work with, but at the same time it would not spoil the performance of the application. As a suitable structure, we chose a hash map of the following form:

{
  'layer_0': {
    "0~0": Pixel,
    "2~0": Pixel,
    "4~0": Pixel,
    ...
  },
  'layer_1': {
    "0~0": Pixel,
    "4~0": Pixel,
    "8~0": Pixel,
    ....
  }
  ...
}

Now, at any time in debug mode, the hash map with the current state of the application was available in the console, while we did not lose anything in performance, since retrieving an element from a hash table using a unique key is performed in constant time.

At the first level, the hash map is divided into layers, where layer_0 is the original (lowest) layer, which consists of squares with the smallest side – 2, and layer_${n} with the highest value is the final layer, which is visible to the user immediately after initializing the program.

At the layer level, the key is the coordinate of the top left corner of the “pixel”. The value is an instance of the Pixel class, which we will get to a little later in the article. Using hash maps, we solved the problem with the opacity of what is happening in the program. Now at any moment we can see the current state of the application.

The Pixel class is used to represent a fraction of an image. It represents a separate image fragment that can be divided into smaller parts. An instance of this class contains all the necessary information about the faction – color, link to child elements, “weight” of the element for calculating the speed and percentage of opening the picture, etc.

Another important task was to calculate the speed of opening the picture and the current percentage of its opening. To do this, as indicated in the previous paragraph, each instance of the Pixel class was assigned its own “weight,” which was subtracted from the total weight of the blurred image upon successful division of the element into smaller fractions or its complete removal from the svg. The percentage of open images was also calculated (this was one of the business objectives according to the approved creative). To do this, when initializing the application, we ran an interval for a second, which counted how many elements were expanded during a given period, and displayed the current speed in pixels per second.

After the program was written, we started testing, and it quickly became clear that our implementation had three significant drawbacks:

Full disclosure takes a lot of time; the user is unlikely to spend several minutes to complete such activity on an advertising page.
In areas of the image that are not very contrasty, finding pixels that are not yet blurred can be quite difficult. Especially if we are talking about a plain background.
In the original implementation, we made it so that the final layer (the original image) after 100% blurring was the first layer of the processed image, that is, an image of twice the quality of the original.

We solved the first problem by adding a condition according to which, upon reaching a certain percentage of disclosure, the remaining factions are also revealed, fully displaying the original picture. We also reduced the number of layers, making the minimum fraction with a side of 16px.

To make the faction layer more visible, we added a shadow to the svg element within which the elements are meshed.

As for the last problem, the solution was very simple; instead of the bottom layer of the processed image, we began to display the original image. What appears to users as one picture is actually made up of 2 pictures, like the scratch line on a lottery ticket. The bottom layer is the desired picture. The top one is its generated svg representation. What appears to users as just a “pixelated picture that blurs out” was a rather interesting logical exercise)

View code: https://github.com/Onsh-a/d3_puzzle

We'd love to hear your feedback in the comments!