From Albumentations to Image Search
By this link dataset search application open images (1.7M pictures).
Upload your picture – get 5 similar ones.
Questions:
Or maybe not a website, but an API?
Or maybe not 5, but 1000?
We are very sensitive to data privacy. Is it possible on our hardware?
And maybe not on open imagesbut on another dataset?
Is it possible to display not only pictures, but also segmentation masks, boxes and class labels?
Will it work on a billion pictures?
Is it possible to do a search not with a picture, but with text?
The answer is a firm yes.
Although it is not clear with other domains. The current search works on natural images and new models are needed for medical and satellite. I don’t have them at hand, although there would be a desire and this can be done.
If someone needs such an image search – write in the comments or to me at LinkedIn – we will understand what functionality to add.
As long as they come to mind:
On public data: the company has a model, its data is not enough. We uploaded pictures on which the model performs poorly, got similar ones, marked them up, and added them to the training data. The quality of the model has improved.
On your data: exactly the same as above, but on unlabeled company data.
On your data: can be used for faster markup. For each picture, a cluster of similar pictures was displayed. I looked with my eyes – all raccoons. I tagged 20 pictures at once. 20x Markup Speed.
To your data: Check markup quality. We evaluate the quality of the markup of segmentation masks or boxes not on one picture, but on a cluster of similar ones. According to one, it is not always clear what is well marked and what is not, and when you compare several, the flaws in the markup are striking.
If someone has ideas or just thoughts out loud to whom such functionality may be needed – do not hesitate to write. I will be glad to ideas of any degree of savagery.
For logging – the logic by which I got to Image Search.
In my spare time I participate in a project Albumentations.AI is an Open Source library for data augmentation.
I wrote about how it developed on Habré – The Birth of Albumentations.
At the best of times, there were 5 people in the Core Team.
A couple of years ago, working on the library turned from something pleasant and for oneself into support.
At that moment, Sasha Buslaev broke away from the team, after whom, in fact, the library was named. I also began to participate less in the project – support, and so everyone at work is engaged for money, but here in their free time and for free.
And there are four of us left.
When the war began in February, Zhenya Khvedshchenya fell off the team. Zhenya lives in Odessa, rockets began to fly into neighboring houses and there was no time for the library.
And there are three of us left.
War is a vile business, so not only Zhenya, but also us, she unsettled. The library has not been touched for several months.
Moreover, we ourselves have not trained the models for a long time, and when we trained the current functionality, it was enough for our eyes.
That is, by May 2022:
hung on GitHub 300 issues
with 10,000 downloads per day and 10,000 stars on GitHub
That is, on the one hand, the library has good popularity, and on the other hand, the project had every chance to die.
Idea one: find someone to take over the project. Exactly the same problem got out there – while the eyes are burning, a person is working on a project, but at some point they will fade away.
Idea two: inject new blood into the core team. We invited one cool developer who created quality pull requests to join the Core Team. He politely refused – he has his own open source library that needs to be developed.
Idea three: find $5,000 a month, hire one person to work full time.
I asked around on LinkedIn, and with startups I know, how can you even find money in such a situation?
There are two options:
Collect donations.
Build Company Based Albumentations in Mode open corethat is, the library is free as it was, but on the side of it, so that SaaS, or the functionality that companies need: privacy, security, and so on.
By the way, it turned out that I, and my friends, think differently about the value of a library.
I:
Speed.
Convenient functionality.
A wide range of transformations for different data domains.
Startups:
I spent several weeks talking with those who use the library.
Collected feedback (blog post on Habré: Albumentations: Feedback) and tried to find ideas for business. So I didn’t find an idea for a business.
I think the main reason is that developer tools are a very bad business.
Much better tools for developer collaboration, and even better for multi-team collaboration.
When I asked questions, I tried to look wider.
Why do we need augmentations at all? Because there is little relevant tagged data. How can such a problem be solved?
In practice, if possible, everyone solves this problem with additional markup.
Often there is a sea of unlabeled data and you need to understand which of them to use.
It is necessary to resize those pictures that “similar” to .problematic ones.
So the word “looks like” is poorly defined.
It could be:
How the model works. Pictures A and B are similar because the XXX model works poorly or uncertainly on them. Moreover, the model can perform poorly in these pictures for various reasons.
color distribution. I had a case when the training data was collected in the summer and the model worked well, and then autumn came, everything became darker and the model swam.
Semantics. a raccoon here, a raccoon there.
Or any other distance metric.
These are all different cases and which one to use depends on the specifics of the task.
The first case is called Active Learning, and I think almost all ML engineers did it by hand. In it, by the way, augmentations are organically used.
Right now, I want to understand where to tie this image search.
Therefore, for those who read the call up to here – poke into the application, think about where it can all be screwed, and unsubscribe.
Hands reach out to cut something down, it remains to understand what. 🙂
PS Write better on LinkedIn
