Ahead of Halloween, full-stack developer Wes Bose in his Twitter asked other programmers to share their horror stories about webdev. Here are some of them…
My first job and my first contribution to production. I need to fix a client-server utility. Its function is simple: we have a homemade device from a router and a PC. The router has a CLI and should show information about the state of the PC. Things like systemctl, top, free. The command is transmitted as text, executed, and the result is received as text. However, this was done in the most horrific way possible …
CLI commands are specified in XML. They invoke a Bash script with specific arguments. The script creates a PIPE in the file system and passes commands to it. The server is written in C (that’s exactly how the server makes requests to the client, not the other way around). All variable names on the C-server consist of one letter (x, a, b, q) or have a general form (mem, data). It also has a bunch of pthreads writing pipes and polling them on a filesystem with completely ZERO thread control. No mutexes, nothing like that.
So, the C-server thread polls the first pipe and executes the script that opens the SSH session with the PC and executes the required command. The result is placed in one more pipe in the file system. The C server has another thread that polls this pipe. And a thousand related problems. For example, he cannot find out in any way whether the result is complete, and does not wait for more data to come through the pipe. As soon as he reads something, he immediately displays it. Because of this, many partial results were obtained, and the remaining parts of them were glued to the result of the next command …
But that’s not all. C code was strewn
fprintf(s)… If you are familiar with the family of functions
printf, then you realize that this is just asking for a buffer overflow. And they visited us. Many times.
Let’s take another look at the architecture / stack of this magnificent invention:
> sh script
> pipe calling pthread
> script calling pthread
> SSH script
> SSH session to connect to PC
> PIPE 2
> another pipe reading pthread
I replaced everything after the CLI with 30 lines of Bash using ncat.
The previous project sent all bug reports as emails to developers. A small mistake spammed us so many emails that our email service provider nailed our account. For half a day, no one could email clients. We spam ourselves about 5,000 emails per minute.
I’ve created a dark mode for our site, but haven’t added default values for the CSS variables. We were supposed to add the CSS theme file in one place and everything should be fine. It turned out that in another project we did not include the theme variables file and everything turned white. Fortunately, the problem was caught in preparation for production.
In my graduation project, I was really annoyed by its size, so I reshuffled the files.
Created one of the weirdest git commands, which will re-recommit all commits from the beginning.
There is nothing wrong with that, but all these commits scared my team.
About 10 years ago, I made a typo in an anchor tag variable at a Top 5 e-commerce company. Let’s say its address is ABC dot com, and all anchor tags were at AC dot com. The bug has spread to the daily / weekly portion of the site. It took a couple of hours to fix.
Quite expensive came out “oops”.
Newspapers leave important news that may happen to be published as soon as the event happens. When Gabriel García Márquez was very ill, they had already left the news of his death on the home page. A friend of mine made a mistake in the conditional and got posted!
I once fulfilled
npm publish from the wrong folder and pushed the private working repository into npm as a public package. Fortunately, there is a team
npm unpublishif noticed quickly enough (within 72 hours). Otherwise, he will stay there forever …
Once I reinstalled the system on my computer. I thought I had created a backup of my code, but it turns out that it is not. A month of work was lost. There is also a good side to this: I think the new code is better.
I ran a competition for a radio show. Created a form through which users could submit answers to weekly questions. There was an apostrophe in the answer to one of the weekly questions. Not a single record has been created this week …
Not my story, but also scary: the dude who worked with me made an update to the database without the “where”, which is why every user who subscribed to the courses was selected the same course. I spent a whole morning CALLING everyone to get them on the course they wanted, and that dude said he was sick.
In my very first weeks as a developer, another newbie and I removed all SSH public keys from the machine running one of the most critical services.
I also mixed up the flags
false and disabled an advertising campaign with a 50% discount for all clients.
I wrote a regex function to validate the social security number before hashing it to the database. However, before performing the verification, I forgot to reverse decrypt, which is why almost 10 thousand patient numbers were deleted from the database in production. Spent the next two days recovering and reinserting.
At the very beginning of my career, I once accidentally connected via Filezilla to the wrong server and uploaded someone else’s website to a working domain, and then went to rest. After a few hours, I saw that the changes I made were not reflected on the site, and I realized what I had done.
Accidentally turned off the server on the first day of work in the office. I didn’t even realize it until I turned on the switch again and the server made a boot sound, while the technical director turned around and asked: “Was that a server?” The second day at work was much better … It couldn’t have been worse.
In the early 2000s, I wrote a website for money. I debugged the contact page, thinking that I had disabled the delivery itself. On that day, the client received a bunch of emails from “Adolf G.”, “Joseph S.” etc. A few hours later, he politely asked me to stop. Oops.
At first, on the client’s side, they thought they were being attacked by some angry psycho, because the body of the letters consisted of vulgarity. Then it dawned on them that someone had reached their website and they contacted me. Apologies were made and lessons learned.
New database for me. I was working on troubleshooting push notifications and in the process I thought there was a difference between a local system and a production system. I sent a dozen test push notifications with an increasingly growing degree of irritation …
It turned out that messages were received not only by me, but also by the whole team. I was told that test messages were received by everyone on the team. Pancake! Well, okay … The company is technical, everyone understood everything.
But then tweets and emails from investors began to arrive – messages were transmitted not only within the team, but also to the entire user base.
That was the last time in my life I wrote abusive / joking / dismissive test messages of push notifications.
“Please make the checkboxes round, but there are multiple options to choose from.”
In 2005, I was working on a web application for a chain of DVD rental stores — it was moving from the client / server applications that were required for dial-up to a single online application. On launch day, I discovered that the central database synchronized from the client databases was completely destroyed. No one has tested data integrity.
Employees from branches called me: “We see people from other cities in our system, but ours are not in it”.
I spent a day connecting via VNC to local computers, creating database dumps using local MySQL admins, and manually building a central database. And all this after the painful two days of putting the system into operation.
In those three days, I listened to David Bowie’s Let’s Dance seventy times.
I once updated the user’s password in a MySQL database in production but forgot to add the construct
WHERE, so the passwords of all users have changed. After 15 minutes, we started receiving calls and emails from corporate clients that could not log in.
Rewrote the form to clean up the UI, which prevented the addition of new credit cards on the mobile site of the Fortune 500 company for a month. Probably no one added them from the app because we received no complaints. The problem was noticed by a junior developer, even the QA department did not know about it.
Another time, the client disobeyed me and pushed a broken template onto the running server. They lost 70 thousand dollars in an hour. This time the mistake wasn’t even close to mine, lol.
On hot days at a startup, I created a game called Zombies, Run! Which was quite popular and had several million paying players. Once, having accumulated immeasurable technical debt, I copied and pasted the wrong version of the Django config file and lost the database secret.
Of course I did it in production and I had no backups. Realizing what had happened, I told the CEO about it and just left the office. I didn’t think we could save the company by texting each user “hello, pay again,” and I got tired of the tech debt debate. It turned out that I was right.
The next morning, I lay in bed pondering ways to reduce the entropy of the secret, and found that since I made another mistake and inserted the secret into my terminal, it was in my zsh history all the time. Two negatives make an affirmative.
Comment on the tweet: I once played this game, it had a bunch of weird bugs.
@marcelcutts: Not surprised, I’m a terrible developer.
I had a multi-site system where content could be transferred from one site to another. During testing, I did not notice that I followed the link to production and launched “update content” without specifying the page-id … All pages on each site then consisted of a single word “testing”. This was in the days before version control systems existed and changes were instantly transferred via FTP to the production server.
In my first front-end job, I was asked to design a long list of emails, so I created everything with divs, animations, and flexbox. After about two weeks, the marketing department ran a quality control and said they would do everything from scratch on tables and inline styles in three days.
Cherry on the cake: the design was supposed to have a pixel perfect layout for the Outlook client.
I have worked on a dedicated e-commerce platform for a Fortune 500 company. Hundreds of developers are working on dozens of international copies of several brands, the code pushes nonstop all day.
So imagine my horror …
when i got the message that i broke my whole git branch / environment. It turns out that when doing merge I messed up the branch I’m on (don’t use ‘.’ Guys. Always include branch names explicitly).
I use a Wacom tablet instead of a mouse, because that’s more comfortable for me. One day, I unwittingly dragged a folder to another folder on the production server. There were no records of what I did, so it took another person a long time to figure it out. And I did it twice.
On another rainy Friday night, I accidentally deleted an SQL table.
Of course, then I found out that our backup strategy was broken.
Fortunately, this was before the lazy loading era, and I ended up with one tab open with 400 values, which I then manually inserted.
I’m usually an agnostic, but I remember praying that day that Firefox wouldn’t crash until I took screenshots of all the values on this page!
VDSina proposes secure servers on Linux or Windows – choose one of the pre-installed OS, or install from your image.