I will probably go to hell for it, but this made me laugh out loud. <Via Jason Garrett>
About 7pm yesterday I took the box entirely offline and wiped the main hard-drive. Now, being a cunning little net.monkey, I had all of your files on a different drive (and a backup from a couple weeks back on my work computer too :) so we didn't really lose much. The virus actually only attacks programs (ELF binaries for you geeks) so I wrote a quick program that purged all programs from your directories. (This only effects Bud and Chili since they actually compiled stuff of their own.)
I put an entirely new version of Linux on the server, replicated what configurations I could, and rewrote the rest from scratch, got the new OS and the old drive working together (quick, is it /dev/hdb2 or /dev/hdc1 ???), and then just beat it with a hammer till it mostly worked. W3ME people we're held up the longest but at this time all services are back up.
There will be another reboot late Sunday when I finish updating to newest versions of some things. And we'll do better about that in the future so we don't get attacked again, promise!
What all this means to you is that you have all you old stuff safe, and some NEW STUFF TO PLAY WITH!
Stats:
Give her the full test and mail me at mark at xodiax dot com if the post system acts weird. No promises and mail is still wonky but I think it all works blog-wise. And yes, I started at 7pm and it is now 3:30am... I wanted to _play_ on the computer tonite, not fix some little shit's mess by doing a full reinstall!
Oh how I love thee Mark,
let me count the ways,
you are so magnificient!
(ok, that's all the creativity I can muster up at this point)
Jade will go off air here in a bit to clean an exploit! We're going to do this at about 7pm now, since I'll need dinner and probably a good cry whilst curled up in a corner first. :) The exploit was via PHP, thank you very much, so we probably won't be installing that again.
What we will be trying to get going is the Image Magic stuff needed to make thumbnails and the picture manipulation features in MT work correctly. *sigh*
And in case you are wondering, it is a new exploit with an old, nasty payload that turns every binary program on the drive into a trojan. A mostly ineffective trojan, and I've blocked its access to the world for now so its teeth are pulled. One thing it does tho is prevent programs from shutting down properly; this explains the weird network reset issue, the shutdown hang from last week, and the webserver failures from last night. Lovely.
We're experiencing weirdness again. I may have to do some sort of reinstall or possibly sacrifice a chicken. I'm hoping it just has computer indigestion. Assuming this site is up and you can read this, go visit Umamitsunami, and be amused with her gaming experiments. The Trance Vibrator for the game Rez is really fucking weird. The game rules, and one women we showed it to immediately turned to her boy and growled, "you need to get one of these!"
Basically, the server was dead because I suck at server administration. When I rebooted the box a couple weeks ago, something didn't start up right. When I noticed it, I fixed the system and rebooted just to make sure everything was fresh.
Of course, I didn't make sure that things were set up to shut down properly so the box got freaked out when shutting down and just plain refused to do anything at all. I should have had one of the NOC guys at work reboot it, but I thought that it had gotten freaked out AFTER rebooting so I never guessed just flipping the power would be all that was required to fix it. Sorry, hostees.
It is once again answering for all IPs and all the sites it is supposed to be handling. It seems to be acting a bit sluggish but that might just be my imagination. Let me know if you see it doing something weird, try mark at xodiax dot com if my regular mail doesn't work!