new SAM neural network that changes the rules of the game in computer vision

Traditionally, image segmentation requires a large amount of labeled data and specialized models for each type of object. However, what if there was a model that could segment any object in any image with a single click? And what if this model could adapt to new objects and images without additional training? Sounds fantastic, but such a model already exists! It is called SAM (Segment Anything Model) and was developed by researchers from Meta AI, one of the largest companies in the field of artificial intelligence.

SAM is an impromptu segmentation system that can accept different types of queries (text, dots, boxes) and generate multiple mask options for ambiguous queries. SAM also has the ability to zero adapt to unfamiliar objects and images without the need for additional training. SAM was trained on the largest segmentation dataset (SA-1B), which contains over 1 billion masks on 11 million licensed and private images. SAM is also designed to be efficient and flexible for high quality and segmentation speed.

In this article, you will learn more about SAM and its capabilities, and look at how it can be used to advance computer vision and other systems and applications.

Main features of SAM

One of the key features of SAM is its ability to adapt to new objects and images without the need for additional training. SAM is trained on the largest segmentation dataset (SA-1B), which contains over 1 billion masks on 11 million licensed and private images. Through this, SAM has gained a common understanding of what objects are and how to extract them. This allows him to segment any object in any image, even if he has not encountered it before. For example, a SAM can segment a dinosaur or a spaceship even though it has not seen such objects in its dataset.

Another important feature of SAM is its ability to generate multiple mask options for ambiguous requests. Sometimes a query can be fuzzy or obscure, such as “highlight flower” in an image with multiple colors. In such cases, SAM can offer several possible masks for different interpretations of the request and allow the user to choose the most suitable one. This is an important and necessary feature for solving the segmentation problem in the real world.

SAM training: how the largest dataset for segmentation (SA-1B) was created using a model in a data annotation cycle

For SAM training, Meta AI researchers developed an efficient model in a data annotation loop that allowed them to create the largest segmentation dataset (SA-1B). The dataset contains over 1 billion masks across 11 million licensed and proprietary images. The dataset covers a wide range of image domains such as underwater, cell, sports, etc.

The data annotation cycle consisted of the following steps:

• Collection of images from various sources such as Flickr, Instagram and YouTube.

• Pre-training of the SAM model on a small subset of images with masks obtained from other datasets or using semi-automated methods.

• Using the SAM model to interactively annotate new images using different types of queries (text, points, frames, polygons).

• Update the SAM with the new masks and repeat the cycle.

After annotating enough masks with SAM, the researchers used its advanced uncertainty-aware design to automatically annotate new images. To do this, they presented SAM with a grid of dots in an image and asked it to segment everything that is in each dot. This method allowed them to obtain over 1 billion masks from 11 million images.

The SA-1B dataset is a unique resource for research and development of segmentation models and computer vision in general.

Efficiency and flexibility of SAM: how the model was designed and optimized for high speed and quality of segmentation

SAM is not only powerful, but also an efficient and flexible segmentation model. It is designed in such a way as to provide high quality and speed of segmentation under different conditions and tasks. To do this, SAM uses the following features:

• The network architecture of SAM is based on the CLIP model, which uses contrast learning to match text and images. SAM extends CLIP by adding an additional decoder for generating masks. The decoder has several levels of detail, which allows it to adapt to different sizes of images and objects.

• SAM uses a technique called prompting to adapt to different kinds of queries and segmentation tasks. Prompting is when the model is given a special text request that tells it what to do. For example, the query “select dog” will tell the model to segment the dog in the image. Prompting allows the model to transfer its knowledge to new domains and tasks without additional training.

• SAM takes into account uncertainty when segmenting objects. Sometimes a query can be fuzzy or obscure, such as “highlight flower” in an image with multiple colors. In such cases, SAM can offer several possible masks for different interpretations of the request and allow the user to choose the most suitable one. This is an important and necessary feature for solving the segmentation problem in the real world.

• SAM is optimized for high segmentation speed. It uses efficient image processing techniques such as quantization and compression to reduce the amount of data and computation. It also uses a technique called image embedding precomputation, where the model precomputes vector representations of images and stores them in memory. This allows the model to instantly generate masks for any query without reanalyzing the image.

These features make SAM an efficient and flexible segmentation model that can operate in real time and adapt to different conditions and tasks.

Conclusion

This is a new breakthrough in the field of computer vision, which opens up many opportunities for research and development of segmentation models and other tasks. SAM can be used in various systems and applications that require the selection of objects in images or videos. For example, SAM can help with image editing, metaverse content creation, medical image analysis, face and scene recognition, robot training, and more. SAM can also be integrated with other models such as CLIP or MCC to create multimodal systems that can understand and process text and images.

SAM is a step towards creating a fundamental model for computer vision that can serve as a universal tool for solving all sorts of problems.