How to use superpixels to improve data annotation

To the start flagship course in Data Science we talk about superpixels that improve data segmentation, and introduce readers to the Kili Technologies platform, in whose tools they are implemented. Below the cut are details and a link to the code for a quick implementation of the SLIC segmentation algorithm.

To understand what superpixels are for, let’s first figure out what “segmentation” is. Image segmentation is recognition of the shape of objects. When segmenting an image, each pixel is assigned a class. Take, for example, the image segmentation problem for the circle and square categories.

Unlike drawing bounding boxes or dots, the goal of image segmentation is not only to learn about the existence of objects, but also to determine their position, size, and exact shape. Using the Kili interface, let’s segment, for example, a car:

Input ML models are better off getting pixel-level segmentation rather than bounding boxes or dots. But creating freeform polygons around objects is much more difficult. The Kili platform allows you to annotate any complex rectangle.

In industrial volumes, semantic segmentation is used only if objects of a very complex shape are being annotated. Bounding boxes often have a better balance of annotation complexity and time. This means that the ratio of results to costs is usually better.

To reduce the cost of segmentation, we have integrated interactive segmentation and superpixels into the platform. They allow you to create segmentation masks in less time (as in the old approach or even better). We already have a whole article about interactive segmentation based on deep learning as an annotation helper method.

In this article, we’ll talk about superpixels.

What are superpixels?

Let’s start by defining the pixel. A pixel is the basic building block of an image, a small square of a specific color. The grouped pixels form an image rectangle. At the same time, there are many ways to group pixels: by proximity, color, etc.

By carefully grouping the pixels, you can improve the annotation. How? For example, in the case of a car backlight in the image below, where the backlight is accurately displayed with a superpixel and the light can be annotated with one click:

Why are superpixels interesting?

Among the ways to group pixels are some that really help make annotation better. It can be accelerated by semantic grouping of pixels into superpixels.

In Kili, we created the Brush tool, with which superpixels are selected very quickly. The superpixel interface is integrated with semantic segmentation, which means that if you need to manually correct something, you can do it.

What interests us when using superpixels?

We have already described what should be in superpixels. But it is very important to clarify the points that determine their appearance:

• Semantics. Superpixels should have semantic content, which is expressed in observance of the sequence of shapes, color uniformity, etc. It should be possible to easily define boundaries by ideally choosing superpixels; so as to avoid manual adjustments.

• The expediency of forms (compactness). Superpixels should have similar shapes. This does not mean that all superpixels should have the same number of pixels, but they should not be too small or have very complex shapes with thin areas.^one…

• Color gradients. Superpixels should be separated by perfect color gradients that define the boundaries that will be best used by machine learning models. This aspect is especially interesting, because computers define such gradients better than people, so superpixel boundaries are more efficient and accurate.

• Speed. Computing superpixels can be computationally intensive. It is important to keep them within a reasonable time frame, avoiding a waste of time so that they turn out to be faster than human annotations.

• Various resolutions. It should be possible to resize superpixels. Sizing can be tricky, but you can estimate the average size or the total number of superpixels. By changing the resolution of the superpixel, you can make annotations more accurate. By changing the resolution, you can annotate shapes of any size.

• Conformity. When the number of superpixels increases when the resolution is changed, borders should never be removed, but new ones should be added instead, otherwise some of the work will be lost when switching resolutions, which is fraught with problems: not all algorithms can prevent losses.

How are superpixels calculated?

We will not go into details, but describe only in general terms the main methods of calculations. There are two main classes of superpixel algorithms: graph and cluster. Graph methods interpret each pixel as a node in the graph, and its edges as anchors. The pixels are then combined into superpixels. Cluster methods gradually refine pixel clusters until a given criterion is met.

We use the most modern graph methods, constantly improving algorithms and superpixel speed, and adapt them, choosing the most suitable for each image. Want to experiment with superpixels? Then get started with a quick implementation of the SLIC algorithm in Python.²…

Superpixels are especially good for tasks where objects are proportionally small compared to the rest of the image and have well-defined colors, as shown below. In addition, there is an interactive segmentation tool that should work well in most cases where superpixels are not ideal.

How to use this in the Kili interface

Superpixels are easy to work with on the Kili platform. Through a graphical interface or API, you create an image project. Then go to the project settings and add the task of semantic image segmentation with the superpixel enabled:

Then click the start labeling button to go to the label creation interface. And click superpixels to generate superpixels:

Then select the category and superpixels by dragging or clicking on them. You can also change the size of the superpixels, controlling the level of detail and the accuracy of the display of certain parts of the image:

Conclusion

We strongly believe that data quality is at the heart of the machine learning revolution. Making the most of the models through good data is the mission of the Kili platform.

The use of various tools, such as superpixels, can reduce the cost of perfect pixel segmentation and train models on the best data. More information about the tools can be found in the documentation…

To continue your study of artificial intelligence in order to learn how to solve business problems, you can in our courses: