The C-Brats Forum Index
HomeForumsMy TopicsCalendarEvent SignupsMemberlistOur C-DorysThe Brat MapPhotos

Ouch...server dead for almost 24 hours
Goto page 1, 2, 3  Next
 
Post new topic   Reply to topic    The C-Brats Forum Index -> C-Brat Site News
View previous topic :: View next topic  
Author Message
Da Nag



Joined: 24 Oct 2003
Posts: 2818
City/Region: Port Angeles
State or Province: WA
C-Dory Year: 2008
C-Dory Model: 16 Angler
Vessel Name: Taco
Photos: <a>Da Boats</a>
PostPosted: Sat Sep 26, 2009 1:25 pm    Post subject: Ouch...server dead for almost 24 hours Reply with quote

We're back. Sorry, folks...my bad.

What was intended to me a minor update yesterday, quickly turned into severe ugliness. A patch applied to the server required a reboot, which normally takes a minute or so.

The server failed to come back up...which is an absolute PITA to deal with remotely, as I'm not sitting in front of the computer to see what is going on. My only recourse, was to deal with two levels of tech support, playing a long and frustrating "game" of telephone.

Support is by email only, and to make matters worse, my support contact doesn't do the troubleshooting in front of the computer. Instead, it went something like this:

Me: "Please try this."

Wait 5 minutes to 2 hours...

Support: "I'll forward that to the data center techs."

Wait 10-30 minutes...

Support: "Didn't work. Should we rebuild the server?"

Me: "Hell no! Try this."

Repeat the above for about 24 hours...


Anyway, everything appears to be functioning again, and lessons were learned from this experience. In particular, I put together documentation for the hosting provider, that allows them to bring the server up via a CD when the server won't boot on its own. In this "rescue mode", the server doesn't function normally - but it does allow me to get onto it remotely, and fix things directly without the game of telephone mentioned above...

Welcome back, friends...

_________________
Tux Will, C-Brat Nerd Tux
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Doryman



Joined: 03 Oct 2006
Posts: 3807
City/Region: Anacortes
State or Province: WA
C-Dory Year: 2006
C-Dory Model: 255 Tomcat
Vessel Name: Lori Ann
Photos: Lori Ann
PostPosted: Sat Sep 26, 2009 1:37 pm    Post subject: Re: Ouch...server dead for almost 24 hours Reply with quote

Da Nag wrote:
What was intended to me a minor update yesterday, quickly turned into severe ugliness. A patch applied to the server required a reboot, which normally takes a minute or so.

The server failed to come back up...which is an absolute PITA to deal with remotely, as I'm not sitting in front of the computer to see what is going on. My only recourse, was to deal with two levels of tech support, playing a long and frustrating "game" of telephone.

Support is by email only, and to make matters worse, my support contact doesn't do the troubleshooting in front of the computer.


Bill -- been there, done that -- but fortunately hands-on. Trying to anticipate all contingencies when writing up instructions can be very frustrating. Thanks for your efforts!

What was the date and time of the backup that you restored from? That will let us know how much activity was lost.

Warren

_________________
Doryman
M/V Lori Ann
TomCat 255, Hull #55, 150 Yamahas
Anacortes, WA
Back to top
View user's profile Send private message Send e-mail AIM Address
Comfortably Numb



Joined: 30 Aug 2007
Posts: 146
City/Region: Greenville
State or Province: IL
C-Dory Year: 2008
C-Dory Model: 22 Cruiser
Vessel Name: Comfortably Numb
Photos: Comfortably Numb
PostPosted: Sat Sep 26, 2009 1:40 pm    Post subject: Reply with quote

I understood about 10% of what you said, Bill! But, that's why you guys are in charge. Thanks for all the good work...you done good.
_________________
Mark
Comfortably Numb
Governor Bond Lake
Greenville, Illinois

“For this is how the suicide of nations begins, when sentimentality prevails over sense.”
- Roger Scruton
Back to top
View user's profile Send private message Send e-mail
Casey



Joined: 02 Nov 2003
Posts: 1094
City/Region: The Villages(FL)
State or Province: FL
C-Dory Year: 2006
C-Dory Model: 23 Venture
Vessel Name: "Dessert 1st"
Photos: Dessert 1st
PostPosted: Sat Sep 26, 2009 1:46 pm    Post subject: Reply with quote

I didn't even understand 10% !

On the otherhand, what I DID understand was that I was beginning to shake from Brat Withdrawl; it was not a pretty sight!

Glad we're reconnected with the Brat Community!

THANK YOU, THANK YOU, THANK YOU, THANK YOU, THANK YOU!!!!!

Best,
Casey

_________________
2013 CC23 "Katmai" renamed "Dessert 1st"
2006 CC23 "Katmai" (purchased August 2009)
2003 CD22 "Naknek" (sold May 2008)


Last edited by Casey on Sat Sep 26, 2009 1:48 pm; edited 1 time in total
Back to top
View user's profile Send private message
Da Nag



Joined: 24 Oct 2003
Posts: 2818
City/Region: Port Angeles
State or Province: WA
C-Dory Year: 2008
C-Dory Model: 16 Angler
Vessel Name: Taco
Photos: <a>Da Boats</a>
PostPosted: Sat Sep 26, 2009 1:48 pm    Post subject: Re: Ouch...server dead for almost 24 hours Reply with quote

Doryman wrote:
What was the date and time of the backup that you restored from? That will let us know how much activity was lost.


Didn't have to go to backup - there was zero data loss. Once I had access to the server, I was able to repair the reason it wasn't coming back up on its own.

For the nerds in the audience...the ultimate problem, was a grub failure. In my best attempt at laymen's terms - the code that tells the server how to start, became corrupted by the patch I applied. It didn't affect any of the data on the server - just the startup process. Once repaired, the server came back up unaffected by the problem.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
BrentB



Joined: 15 Jul 2006
Posts: 4419
City/Region: Greenwood
State or Province: IN
Photos: BrentB
PostPosted: Sat Sep 26, 2009 1:51 pm    Post subject: Reply with quote

thanks again for your hard work and I added something to the Tip jar
_________________
Brent Barrett
Back to top
View user's profile Send private message
Jack in Alaska



Joined: 17 Aug 2004
Posts: 1190
City/Region: Anchorage/Ninilchik
State or Province: AK
C-Dory Year: 2005
C-Dory Model: 26 Pro Angler
Vessel Name: HIGH TIDE II
Photos: HIGH TIDE II
PostPosted: Sat Sep 26, 2009 2:05 pm    Post subject: Reply with quote

DaNag................

You are just great at keeping us "fat fingered" C-Brat junkies up and running. Thanks for your expertise and your effort in doing that for us. I am only a 5% understander. I also suffered withdrawal pains but am over it now.

Jack in Alaska

_________________
On the HIGH TIDE-II, wife Carolyn and I.....Another summer fishing on the HIGH TIDE II in the Cook Inlet at Cape Ninilchik, Alaska.

HIGH TIDE-II; 2005 26' ProAngler; 2003 200 Honda / 2009 9.9 Honda high thrust
No. CD026021I405; AK-5008-AK
MSSI No. 338143486(cancelled)

HIGH TIDE; 1983 Angler Classic 22'; 90 Honda/ 9.9 Tohatsu-sold 2009 to son Dan (flatfishfool)
Stolen & stripped in Aug. 18
Bare hull & trailer sold in Nov.


Last edited by Jack in Alaska on Sat Sep 26, 2009 2:14 pm; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail
colobear



Joined: 23 Jan 2005
Posts: 2154
City/Region: Denver
State or Province: CO
C-Dory Year: 2006
C-Dory Model: 22 Cruiser
Photos: C-Cakes
PostPosted: Sat Sep 26, 2009 2:06 pm    Post subject: Reply with quote

Thanks Bill. We sometimes need something like this to help us remember all the hard work you and Mike do to provide this great site...and how important this site is to us.

Thank you!

_________________
Patti and Barry
formerly C-Cakes, now
rving around N. America
Back to top
View user's profile Send private message Send e-mail
Lucky Day



Joined: 10 Oct 2006
Posts: 215
City/Region: Churchton
State or Province: MD
C-Dory Year: 2004
C-Dory Model: 25 Cruiser
Vessel Name: Lucky Day
Photos: Lucky Day
PostPosted: Sat Sep 26, 2009 2:09 pm    Post subject: Reply with quote

Losing c-brats.com is a little like the shock of losing electricty. Wow. Glad that's over. Had to click on "Donate Now" for the effort that y'all put in . . .
Back to top
View user's profile Send private message
Da Nag



Joined: 24 Oct 2003
Posts: 2818
City/Region: Port Angeles
State or Province: WA
C-Dory Year: 2008
C-Dory Model: 16 Angler
Vessel Name: Taco
Photos: <a>Da Boats</a>
PostPosted: Sat Sep 26, 2009 2:14 pm    Post subject: Reply with quote

Thanks much for the kind words.

But, be careful with that Tip Jar. I just might realize, that screwing things up around here can be a money maker. Heck...maybe next time, I'll take a week to fix things - just might get enough for a TomCat. Shocked
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Doryman



Joined: 03 Oct 2006
Posts: 3807
City/Region: Anacortes
State or Province: WA
C-Dory Year: 2006
C-Dory Model: 255 Tomcat
Vessel Name: Lori Ann
Photos: Lori Ann
PostPosted: Sat Sep 26, 2009 2:31 pm    Post subject: Reply with quote

Da Nag wrote:
Thanks much for the kind words.

But, be careful with that Tip Jar. I just might realize, that screwing things up around here can be a money maker. Heck...maybe next time, I'll take a week to fix things - just might get enough for a TomCat. Shocked


OTOH, given enough tips, you could afford to host the site where you have hands-on access!

Warren
Back to top
View user's profile Send private message Send e-mail AIM Address
BrentB



Joined: 15 Jul 2006
Posts: 4419
City/Region: Greenwood
State or Province: IN
Photos: BrentB
PostPosted: Sat Sep 26, 2009 2:33 pm    Post subject: Reply with quote

Da Nag wrote:
Thanks much for the kind words.

But, be careful with that Tip Jar. I just might realize, that screwing things up around here can be a money maker. Heck...maybe next time, I'll take a week to fix things - just might get enough for a TomCat. Shocked


sorry I am a bad tipper
Back to top
View user's profile Send private message
BRAZO



Joined: 29 Jul 2008
Posts: 650
City/Region: Full-time Travel
State or Province: NV
C-Dory Year: 2008
C-Dory Model: 22 Cruiser
Vessel Name: Meyer Meyer
Photos: BRAZO
PostPosted: Sat Sep 26, 2009 3:05 pm    Post subject: Reply with quote

Quote:
What was intended to me a minor update yesterday, quickly turned into severe ugliness. A patch applied to the server required a reboot, which normally takes a minute or so.


As a Computer Programmer for 25 years - I know the feeling, been there done that - but in my case it would take down the mainframe for 2500 employees for a bit, not good.

Very cool that c-brats have someone like Da Nag with the knowledge and the willingness to get it done.

Many thanks.

_________________
2008 CD22 Twin 50hp
Full-time Travel - Sprinter4x4, International, C-DORY 22
Back to top
View user's profile Send private message
MyCattMaxx



Joined: 16 Sep 2009
Posts: 9

C-Dory Year: 2009
C-Dory Model: R-25 Tug
PostPosted: Sat Sep 26, 2009 3:15 pm    Post subject: Reply with quote

Yup, I kept trying to get on here and TugNuts.
I assume you both are on the same server?
Back to top
View user's profile Send private message
Da Nag



Joined: 24 Oct 2003
Posts: 2818
City/Region: Port Angeles
State or Province: WA
C-Dory Year: 2008
C-Dory Model: 16 Angler
Vessel Name: Taco
Photos: <a>Da Boats</a>
PostPosted: Sat Sep 26, 2009 3:17 pm    Post subject: Reply with quote

MyCattMaxx wrote:
I assume you both are on the same server?


Yup.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    The C-Brats Forum Index -> C-Brat Site News All times are GMT - 5 Hours
Goto page 1, 2, 3  Next
Page 1 of 3

 
     Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum



Page generation time: 0.0902s (PHP: 69% - SQL: 31%) - SQL queries: 33 - GZIP disabled - Debug on