Server outage for multiple days

Status
Not open for further replies.
S

sub.mesa

2[H]4U
Joined
Feb 16, 2010
Messages
2,508
Hey Kyle/HardOCP admins,

I was curious about what happened that caused your downtime of past days, and whether you lost any posts?

Cheers,
-sub
 
Nice job bring it back up. However with the lack of hardforum I gotten sick.
 
52 hours of maintenance that was unannounced?? Umm sure, ok...

Yeah I was missing the site too. Glad it is back online now.
 
WTF? Even Slashdot is having issues.

I can't browse HardForum very well right now without doing an SSH Proxy Tunnel to my dedi ... new servers and not fully configured? :S
 
i have a pretty good idea why a lot of vbulletin powered forums are going down for maintenance.. but if kyle/forum admins want to post why i leave it to them instead of starting speculation..
 
See, it's all Steve's fault.
Everybody - STEEEEEEEEVE!!!!!!!!!!!!!! :mad:

jk :D I did start having withdrawals though. No hardforums makes me a sad panda.
 
Astrodave said:
52 hours of maintenance that was unannounced?? Umm sure, ok...

Yeah I was missing the site too. Glad it is back online now.
Click to expand...

Right before it went totally down. There was a message on the site from AnthonyG saying that it would be down for a few minutes.... Oops. :eek:

However I understand since I admin servers myself. Sometimes a simple fix and reboot turns into a full rebuild and restore from backups. Shit happens and overall since the new hardware got figures out a while back, the site has had awesome uptime.


And another reboot message as I tried to post this...
 
The forums were hacked.

We have been hacked three times in 13 years. We are not sure the motivation this time, but it was not "malicious." In the past it has been spammers/phishers using our resources to send out their mails. This time, like before, nothing was damaged, there were simply processes running that were invisible to us and using all the box's resources. Box is a dual Quad Xeon with 16GB of RAM. It took them at least 3 days to get in (from what we can see in the logs), and these guys were very very slick. Have to give them kudos on their talents. I am sure they are making plenty of money with their skill set.

From our past experience with this, we have learned that it is best to start from ground zero in the box that was compromised. So we are doing a full OS and vBulletin install. It has been a couple of years since we did this so there are some upgrades to be made as well.

We have put a measure in place to safeguard us from downtime like this in the future. Sorry for the inconvenience.
 
Kyle were you using the vbulletin version that had the recently announced flaw in it? version 3.8.6 i think it is..
 
Bahamut said:
No need to apologize to any of us, we're all just happy to see the forums back up and functional.

We, and I'm fairly sure I speak for everyone, appreciate the efforts the Admins, the Mods, and you yourself put into this place.

Thanks...
Click to expand...

I agree with this post, thanks to all you guys who worked hard on bringing the forums back to life.

I did find it a little weird that the servers go down when the biggest game of the year launched on Tuesday (Starcraft 2).
 
at first I thought it was my work blocking it...then I thought ip change + dns propagation....

I went a bit crazy with no [H] in my life..
 
The forums were hacked.

We have been hacked three times in 13 years. We are not sure the motivation this time, but it was not "malicious." In the past it has been spammers/phishers using our resources to send out their mails. This time, like before, nothing was damaged, there were simply processes running that were invisible to us and using all the box's resources. Box is a dual Quad Xeon with 16GB of RAM. It took them at least 3 days to get in (from what we can see in the logs), and these guys were very very slick. Have to give them kudos on their talents. I am sure they are making plenty of money with their skill set.

From our past experience with this, we have learned that it is best to start from ground zero in the box that was compromised. So we are doing a full OS and vBulletin install. It has been a couple of years since we did this so there are some upgrades to be made as well.

We have put a measure in place to safeguard us from downtime like this in the future. Sorry for the inconvenience.
Click to expand...

I assume the password database and/or any other parts of the forum that would make people worried, etc. did not get touched in the process?
 
Last edited:
AVT said:
I assume the password database and/or any other parts of the forum that would make people worried, etc. did not get touched in the process?
Click to expand...

as far as I know passwords are encrypted so there isnt really a way to retrieve them if you hack in. at least I think that is the case.
 
Thank God you're back up. I was in the middle of uploading some pics of my new rig and the site disapeared. I was like WOW...I did something.:D
 
Eriksrocks said:
Were you running on 3.8.6 at the time you got hacked?

EDIT:

^this
Click to expand...

No, they werent, as I pm'd kyle about the flaw when the news hit, I checked using firefox to see what version H was using and they were still on 3.8.2 at the time, whether that was also affected I dont know.

Bunch of dicks hacking H.
 
I assume the password database and/or any other parts of the forum that would make people worried, etc. did not get touched in the process?
Click to expand...
No worries for me, even if teh database is encrypted. All my passwords are uniquely [H]ard with KeePass. :D
 
Hmm, now every single time I come here I have to login. It just ignores the remember me checkbox. Worked fine before the outage too. Anyone else having that?
 
I would like to know if there is a possibility our passwords were compromised.
 
I'm glad that the H is back up. I almost called in sick to work today because I didn't want to have to go all day with no H.
 
if hardforum does their password right, there is no method to get the password.

well they can get the blowfish encryption key.
than try to reverse the passwords that are stored as hash data.

however.

if they can use the blowfish key and make it convert all the dictionary words into the keys and compare the database for the hash.


so people with common passwords are the most likely to get their password figure out.
 
Thank god your back! - Appreciate the site a lot and never have any bad feelings for you doing what has to be done to keep it running smooth. Thanks for both yours and all the admin's efforts!
 
AnthonyG said:
favicon.ico is for your site, not for an individual program, it is placed in the public_html or aka root doc folder of your site. it will take a few for everyones browsers to see it, so be patient, its there.
Click to expand...

err what? They're still using the vbull favicon in the root of hardfourm.com http://hardforum.com/favicon.ico

aka Eriksrocks suggestion is valid, although its the least of their concerns at this point.
 
Yeah... without [H] I had time to head over to go outside... Did you know the sky is blue?
I also had time to visit other forums... *mumbles*stupid fatwallet/techbargains*mumbles* now I'm out couple hundred dollars... although I do have new stuff coming in now :-D

Thanks for all the hard work and not making us rebuild the forum one entry at a time. I'm not sure I could remember all the posts I had to keep up my post count...
So do I need to change my password from password2010 to something else?
 
AVT said:
I assume the password database and/or any other parts of the forum that would make people worried, etc. did not get touched in the process?
Click to expand...

Your information is safe, but if you are worried all you have to do is change your PW. Poof, all solved.
 
I was really traumatized by the recent downtime and I am writing a book about what I experienced when I was forced out into the daylight.
 
NKDietrich said:
I was really traumatized by the recent downtime and I am writing a book about what I experienced when I was forced out into the daylight.
Click to expand...

I could only see shadows IRL. Must be all the time I spend staring at [H]'s black screen. /Allegory of the Cave
 
munkle said:
as far as I know passwords are encrypted so there isnt really a way to retrieve them if you hack in. at least I think that is the case.
Click to expand...
You should look up rainbow tables and the way salt works if that is what you think...
If the box was rooted, then any salt(text or formula) done is probably known also(if this was something they were looking for)...
If the passwords aren't salted then the process of decrypting passwords is so fast it's scary due to pre-made rainbow tables :eek:

As a user, always use unique passwords everywhere, at least that will limit the consequences of a "stolen" password.
 
So [H]ard Forum became [H]acked Forum? :eek:

I was feeling a bit soft without [H]ard. :eek:

Good to see it back. :cool:

Damn crackers! :mad:
 
Status
Not open for further replies.
Back
Top