Devin AI – exposure from Reddit users or 3 reasons why artificial intelligence will not replace the developer

On March 12 of this year, Cognition Labs introduced Devin AI, the world's first, according to the developers, fully autonomous AI programmer. Cognition wrote on their blog, “Devin can plan and execute complex engineering tasks, recall relevant context at each step, learn over time, and correct mistakes.” They also demonstrated the capabilities of AI to the audience using a real example from the Upwork website, which caused a wave of discussions in early spring and made programmers fear for their jobs.

However, it looks like we can breathe easy as the company was recently accused of lying. Users doubt the performance of Devin AI, calling examples from promotional videos fakes and pointing out that the released product is not at all true.

It all started with a comment from one of the users on Reddit under a post discussing a new product in the world of artificial intelligence – Devin AI. The post itself has now been deleted, but the comment can be found at link.

User under nickname minegen88 writes: “This looks like a scam. Like for real? Look at their site…can't they use Devin to make it better???” Also in his commentary, he provides a number of evidence, which we checked and analyzed in detail.

1. Devin AI website

Website Cognition doesn't look very good, to put it mildly. It consists of two pages and at first glance looks more like the work of a student who is just learning the basics of web development. But appearance is a subjective thing, so let's figure out what's under the hood.

The first thing that catches your eye when opening devtools is that everything is done on Webflow, a designer for landing pages. And the blog, which is also on the company’s website, is actually a static page. Indeed, having AI at hand that is tailored specifically for development, why couldn’t Cognition use Devin AI and create a good web application with it?

TO website Users also had questions about Devin AI itself. The interface shown in the promotional video does not correspond to the real one. Moreover, it will not be possible to test the technology yet. You can only request access by leaving your email.

Also Reddit users using React Developer Tools We managed to find out that at first the site was working in dev mode. Thanks to the dev mod, you can receive error and warning messages directly to the console, but performance suffers and the bundle size is larger than in the production mod. It is worth mentioning that today, even if such a problem existed in the first days of launch, the developers (and maybe Devin AI itself) have already fixed everything long ago.

2. Inconsistencies in promotional videos

Since ordinary users do not yet have the opportunity to test Devin AI, let’s take a closer look at the videos from Cognition demonstrating the work of AI.

Let's start with the promotional video that went viral the most and caused a certain stir among developers. In it, Devin takes on an Upwork challenge and succeeds, proving that AI will soon be able to replace humans even in complex tasks. However, an attentive viewer may notice that the task for the AI ​​was not chosen by chance (see search on the video – timecode 3:48), most likely in order to show Devin in the most favorable light. It's okay, of course, this is a very common practice. However, then it is strange that AI is presented to the user as “an AI programmer capable of performing any task.”

Several revelations can already be found on English-language YouTube. One of them is from the channel Internet of Bugs. In it, a programmer with 35 years of experience in development named Karl analyzes the promotional video in detail and finds many inconsistencies and gaps in it. For example, it turned out that Davin AI first creates files that were not initially in the task repository with Upwork, and then it itself finds errors in them and fixes them. To someone who has not looked into the code shown in the promotional videos, it appears that Devin is capable of finding errors in the files that were provided to him. However, we were never shown any examples of correcting code that actually existed in the repository, and not something that was invented in the process by the AI ​​itself.

3. Withholding inconvenient facts

Let's take a look at how long it took Devin AI to complete the same task with Upwork. The first message to clarify the task was sent at 15:24. The last message with a review of the final result is at 21:41. As a result, Devin’s work on the task took a little more than 6 hours. An experienced developer, in turn, completed exactly the same task in about half an hour (https://youtu.be/tNmgmwEtoWE?t=1095&si=GXy78TXkYl1Lstgz).

Of course, the technology is quite new and it makes no sense to expect an ideal result from AI. However, none of the videos mention that generating a significant amount of code takes longer than it might seem to the average person. It seems that Devin AI’s entire PR campaign is a deliberate attempt to shock the viewer, to convince everyone that artificial intelligence is developing faster than it actually is.

The exact same errors and inconsistencies appear in almost every video published by Cognition Lab.

Bottom line. Will programmers be left without work?

Judging by the number of memes people have posted on social media over the past week, definitely not. In the next few years, programmers will not face such a fate. The existence of such AI tools can only speed up the execution of routine tasks, search for and explain errors, and in some places eliminate the human factor. However, models that have been in the public domain for a long time already cope with this very well – ChatGPT-4 Turbo or Claude-3 Opus, access to which (and many other models, such as Midjourney, Dalle, Gemini) can be accessed through Bothub.

Programming now is not so much about writing code, but about understanding the customer’s problem, communicating with the client and the team. This cannot be achieved using only artificial intelligence and completely eliminating human intervention.

Even if the functionality of Devin AI in promotional videos was accidentally or deliberately exaggerated, the technology can be a great assistant for developers. It is doubtful that AI will soon be able to independently solve complex problems, fix bugs and write perfect error-free code, as has been claimed. A virtual programmer cannot yet replace a live developer.

Tell us, do you use AI in development? If so, how often and for what purposes?