Code quality has decreased due to neural networks
A study of the GitClear service showed, that recently the overall quality of code in projects has begun to decline. The main reason for this is the proliferation of neural networks that help write code. The trend is forecast to continue.
What’s happened?
GitClear has collected nearly a billion lines of code from 2020 to 2024. Information was taken from customers (NextGen Health, Verizon) and from open source repositories (Facebook React, Google Chrome). Out of a billion, we selected 153 million lines in which significant changes were made:
- adding new, unique rows;
- deleting rows (it is important that they are not returned afterwards);
- moving lines to a new file or to a new function in the same file;
- updating lines by more than three words;
- replacing the same lines of code in different places;
- copying and pasting code within one or more commits.
The team also studied changes to the code that had already been changed—by this they meant code that had been seriously rewritten within two weeks.
The results showed that Code Churn has increased significantly in recent years. This is the name given to code that was removed or seriously changed within two weeks of being added. These are changes without benefit to the project – which reflect the quality of the code as a whole, indicate instability and the number of errors.
Until 2023, such code was only 3–4% (with a surge to 9% in 2022, when Copilot was released). By the end of 2023, the figure increased to 5.5%, and in 2024 it is predicted to be 7% – this is twice as much as in 2021, before the release of neural network assistants.
The 2024 data is a forecast compiled by GitClear using the gpt-4-1106-preview language model. It was created by OpenAI using quadratic regression and Python.
Analysts also checked how long the code lives without changes. The results show that over the past 4 years, the share of code that was changed within two weeks has increased by 10%. At the same time, the share of code that is changed after a month has fallen (this is associated with the length of the sprint).
According to the researchers, this shows that before the advent of Copilot and similar assistants, developers were more likely to look for fragments in their code that could be improved. Now the dominant approach is “write it and forget it after the end of the sprint.”
What does this mean?
Code quality dropped in 2023. This is due to the widespread adoption of LMSs in general and AI-based assistants in particular. After all, they are focused on writing, not maintaining code. And juniors, unlike experienced developers, pay less attention to this.
An alternative tool will be needed that will check the code and suggest improvements. GitClear itself would be interested in working with specialists who are doing (or are ready to do) similar things.
GitHub survey shows, that the developers themselves feel that the quality of the code has decreased. Thus, in the answer to the question “What should you be assessed on in the absence of artificial intelligence,” “code quality” dropped to second place.
