Otakubell went offline on my birthday (yaaaaay), 7/3/14, due to a security breach.
When you run a server on teh interwebs, there are hacker bots trying to break into it continuously. Like, literally at all times, from every corner of the globe. When I first moved otakubell.com from shared hosting to a VPS, I made sure I secured it as well as I possibly could. And I guess I did a moderately good job, because it took almost two years before a break-in finally occurred – or, at least, before a break-in was noticed.
I’m fairly sure nobody broke into the shell, or the SFTP, those were nailed firmly shut with a 2048-bit RSA key. But it appears a combination of bots finally brute-forced my Postfix password, and began using it as a spam relay.
I had fail2ban configured to ban IPs that attempted to access Postfix – but it only banned them for a month or so. Clearly this was insufficient. I also didn’t have a super-long password, and I never changed it.
Once ANYTHING is compromised on ANY computer system, you have to nuke the entire thing from orbit – its the only way to be sure. I had no way of knowing what else had been compromised, and I wasn’t about to take any chances. I wiped the entire system and used the opportunity to upgrade to Debian 7.5, which I had put off for far too long anyhow. I had a recent backup of site data, so very little was lost, if anything.
There will probably be more downtime, as I am still performing various upgrades and tweeks. Otakubell.com went offline for a few hours last night because it ran out of memory – I need to tweak my Apache2 settings a bit more, it seems.
Comments (0)