Server Hiccup

Muhammad · Nov 7, 2001

Around 3:42PM EST our server suffered a slight hiccup, halted, and rebooted without a clean shut-down... all of our databases open when it happened, and as a result we had some corrupted tables to deal with. Fortunately, we were able to restore everything as of 3:42PM as far as we know. There may be some corrupt data remaining, so if you see ANYTHING that looks odd (broken posts, broken links, etc) please let us know as soon as possible.

Before the shut-down, our server had been running over 201 days nonstop, so we're still maintaining 99.98% uptime. /w3tcompact/icons/wink.gif

Harv · Nov 7, 2001

99.98% uptime.

I know your server is running on a Kubota engine, but even so, you have to change the filters once in a a while. /w3tcompact/icons/wink.gif

Corruption is not tolerated on this board, so don't worry about that. Personally, I'm impressed with how quickly you got 'er back on her feet. Glad you're our Secretary of Homepage Security. /w3tcompact/icons/cool.gif

Boondox · Nov 7, 2001

Nice recovery! The only thing I noticed is all the posts I read this morning were marked as unread when I logged back in this evening.

Pete

www.GatewayToVermont.com

Knight9 · Nov 7, 2001

maintaining 99.98% uptime

What a coincidence, our server here is maintaining 99.98% CPU utilization!!!! Wanna trade???

Kevin

Anonymous Poster · Nov 7, 2001

Muhammad,

I thought I hit the wrong keys when I posted a second time...

Then I thought it was something on my end... so I rebooted and saw about 85 registered users and 64 unregistered users under "Who's Online"... WoW... this is overload...

Muhammad you're fast...!/w3tcompact/icons/smile.gif

GlueGuy · Nov 7, 2001

about 3:42PM EST our server suffered a slight hiccup, halted

restore everything as of 3:42PM

That has to be a world record for a reboot! /w3tcompact/icons/shocked.gif

The GlueGuy

Bird · Nov 7, 2001

all the posts I read this morning were marked as unread when I logged back in this evening.

Me, too!!/w3tcompact/icons/frown.gif I'll never get through them tonight./w3tcompact/icons/laugh.gif

Ibrahim · Nov 7, 2001

Yeah,

We had to restore some of the records from this morning at 5am. (When I run the backup.) It was really touch and go for a bit, and I wasn't sure I would have been able to recover today's ANYTHING (posts, basically)

But the User table in the database was where the corruption was (hence the screwed up "who's online" section) and both my account, and Muhammad's account were deleted. (Two others also lost their account, but they're back.)

So I restored the User table in the database with the one from this morning. But it was either that, or delete all the posts from 5am until 5pm. (That's close to 200 or something.)

It wasn't a total disaster, and you can bet that I'll be on the phone tomorrow asking what exactly caused our server to go down on the 202nd day of uptime. Especially with extremely low CPU load ( 0.30 0.21 0.23 for those who it interests.)

I'll keep everyone informed if I find out anything.

-Ibrahim

billboe · Nov 7, 2001

Hello Ibrahim (or Muhammad),

If it's not a trade secret, what kind of hardware and software (OS, web, etc...) do you run? Curious fellow geeks want to know...

Billboe

MarkV · Nov 7, 2001

Ibrahim,

Although I don’t even know what you are talking about, thanks for fixing it. I have been around long enough to know you are one of the Masters behind our Forum Master. Just wanted to say thanks for all you have done to make TBN such a great site.

Muhammad, I think you should give him a raise. You can add it to Harv’s monthly membership fee./w3tcompact/icons/smile.gif

MarkV