During the month-long attack, we had a few days of respite and managed to drive to a startup party from BloombergBeta. A good way to let off steam before the upcoming sleepless nights.
(Repl.it is a service that allows you to write code in most languages directly in the browser, without installing an additional IDE)
Getting involved in startups, no one tells you that fighting bad hackers is part of the job. The strongest opponents that we had to fight in Replit are dark web hackers. One day we came face to face with an underground group that was doing very bad things on our service. And that’s how it was.
In November 2018, Replit was very small, 4 people, but product usage was growing super-exponentially. Because we provided free cloud computing, we created a lot of technology to run containers cheaply. On Thanksgiving Eve, I received a notification that we were spending $10,000 a day, which was 20 times the norm.
All emergencies tend to happen on holidays – is it a coincidence? I sat down at my computer and found that Replit had forked 10,000 times and was sending an insane amount of traffic. I banned the user and warned the team. We won’t have Thanksgiving this year.
Then another “Santa Claus” came and did the same, and another, and another – we were losing a lot of money. The process was clearly automated. So what did they run in our service? Checked the contents, but there was nothing. What the hell?
Working under pressure is a hard-earned skill. Years of extreme programming competition help when you have to solve complex problems around the clock for 24 hours. Years of participation in eSports. And of course, incident response in startups, all this prepared me for that moment.
As a leader, I had to remain calm and radiate confidence. My team is top notch hackers, and I knew we could do it. Okay, we loaded our magazines, clenched our teeth in anticipation of what lay ahead of us.
The first thing we had to do was…wait. We needed to wait for the next attack so we could debug it in real time. At the end of the attack, we would take a look at the running containers. There was a binary – how did they insert it? Anything else? No, no source code, nothing. Only binary.
Okay, did he go over the net? Over UDP? But we don’t have UDP. A curse.
Just in time for the next attack, we rolled out UDP logging. Looking at the logs, they are receiving a payload from an external server. This server may have been the “command and control” server of the botnet.
The content being sent was probably the working binary we saw earlier. We have launched a network trace. The IP address of the external server. Could it be a “command and control” server performing attacks? We took the IP address and compared it with other logs. Oops, a coincidence! Nice, nice.
What did the CnC server do? Reconstructed from logs: cleans up our sites, creates repls and receives tokens, connects to repls, sends packages to them. Basically, they reverse engineered our protocol to create thousands of repls that did… what?
Okay, we needed to make it harder to programmatically create repls. But before we can do that, let’s take a look at what we can learn from their CnC server. In information security, you instinctively go to put out fires, but to win the battle, you need to get more information. We created Honeypot.
nmap will scan the server for open ports (especially HTTP and telnet). The web server looked like a custom server. Maybe there were some vulnerabilities in it? In other words, we switched roles and went on the offensive – we’ll hack them!
After several attempts: Their server was vulnerable to a textbook HTTP hack.
HTTP servers have been programmed to send files from the current directory. But what if you could create a request to request files outside of the current directory—from virtually anywhere?
But scanning their entire file system would take a long time – they might know we were doing it. So the first file we tried to get our hands on was the “locatedb” file, a cached index of all files on the host.
And bam, it worked on the first try and we got all the files on the system.
The most important files we found:
– bot.c: binary source
– server.js: source for this server
We have requested these files. We looked at the sources. The server was so clownish that it opened a reverse shell in telnet. We killed them PWN. Easy.
But the question remained: “What the hell were they doing?”
“bot.c” gave a clue. They used the Replit infrastructure as a massively distributed botnet. Yes, our infrastructure was powerful enough for that. It was not only bad for us, but for everyone they attacked with us.
It was a big financial blow to us, it destabilized our infrastructure and lowered the quality of our services.
But what’s the worst?
Our software was used for evil, to harm other people, it was completely unacceptable for us.
Where are the CnC hosts? digital ocean. We told them about the problem, but it could take them days or weeks to investigate (eventually, with our help, they reported them to the authorities). However, at that moment we had to take care of ourselves. How could we stop the attacks?
It was a game of cat and mouse. We tried to plug holes in our system that allowed us to programmatically launch replas. And they tried to find new holes. But once we have all their files, perhaps we could learn more about them? Perhaps we could even talk to them?
The most important thing we saw were links to their website – what are they? This is a website that sold network attacks for bitcoins. You pay them, give them an IP address and they use their botnets to attack it. And they figured out how to use us as one of their botnets. Get hurt.
We found a link to their discord server. A bit of social engineering: we applied as clients. And these dummies believed us. From within, we began to receive useful information to combat them. When we fixed the vulnerability, we saw how their clients began to complain, and the attackers tried to find our other vulnerability.
“We’ll be back in 15 minutes”
Damn it, they must have found another one of our holes. We reacted quickly and tried to prevent their next attack. This went on for several days. There was no time for sleep and food. Just a few days of computer battles.
A month went by, and then all of a sudden… they gave up. They closed everything and moved on. We have created an insane amount of software to monitor, detect and stop attacks in a very short amount of time. We won. We couldn’t believe it.
After all that was said and done, they cost us a large sum. But on the other hand, it’s like hiring a pentester to help secure our systems. Not a bad result. Replit has become 1000 times safer.
One day I will tell you about how we fought and defeated crypto kiddis.