Server outage *UPDATED 24-jul-2004*

Posts   
1  /  2
 
    
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 12:10:45   

The online server went down at 4:28 AM (CET) and apparently didn't recover. Manually rebooting it on site (a 70mile drive) brought it back up at 10:58AM (CET).

remote rebooting did not bring it up completely at 8:17AM (CET).

Sorry for this inconvenience.

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 13:17:37   

Update: the box has some malfunctioning hardware. Dell is investigating it.

Expect some more outage during this day. Sorry for this inconvenience.

Frans Bouma | Lead developer LLBLGen Pro
netclectic avatar
netclectic
User
Posts: 255
Joined: 28-Jan-2004
# Posted on: 20-Jul-2004 13:20:39   

The joys of remote hosting your own server. I'm glad i dont have to drive to my server though as it's about 3000 miles away confused

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 13:23:58   

netclectic wrote:

The joys of remote hosting your own server. I'm glad i dont have to drive to my server though as it's about 3000 miles away confused

It's 70 or so, but still a long drive.

The box did ok for 2.5 years, before it died last night it was up for over 100 days. However it is of less quality than the box I built myself before this one. This is a dell poweredge 1650 1U server, very fast, but apparently very fragile too. A remote reboot now worked again, but for how long is uncertain... Apparently Dell can read some system info out of it, I'm now waiting for the utility to do that.

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 17:43:22   

Still in research. The server is staying up now, we hope to hear back from Dell later today.

Frans Bouma | Lead developer LLBLGen Pro
jeffreygg
User
Posts: 805
Joined: 26-Oct-2003
# Posted on: 20-Jul-2004 19:24:19   

Ever think about hosting it at home? I'm sure you don't get a huge amount of traffic (except for maybe downloads) - unless all of our RSS readers are pegging you... it's real easy to host at home and nice since you have direct access to the machine.

Jeff...

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 20:07:27   

jeffreygg wrote:

Ever think about hosting it at home? I'm sure you don't get a huge amount of traffic (except for maybe downloads) - unless all of our RSS readers are pegging you... it's real easy to host at home and nice since you have direct access to the machine.

The server hosts 12 sites, 4 of them are ours, and llblgen.com alone pulls 12GB traffic a month simple_smile

Hosting at home / locally is not really an option because upstream lines with 10MBit or more are really expensive here. Now the server is located in a datacenter, directly connected to the amsterdam internet exchange, airco, airfilters etc. The only problem is hardware service. The server is leased from dell, so they have to fix issues fast, but as you might know, dell is not that fast...

Frans Bouma | Lead developer LLBLGen Pro
Answer
User
Posts: 363
Joined: 28-Jun-2004
# Posted on: 20-Jul-2004 20:17:35   

Honest to god, whenever i have a problem, someone else has a problem, which results in me not being able to fix my problem simple_smile

For instance, last night, i was up till 3 am trying to figure out why in the hell my webgrid was so freakin huge. I needed to look in the forums for databinding issues with infragistics as i remember seeing something about it here. Naturally they were down, first time EVAR.

OH welll, i did figure out the problem... smile

jeffreygg
User
Posts: 805
Joined: 26-Oct-2003
# Posted on: 20-Jul-2004 20:25:31   

Otis wrote:

The server hosts 12 sites, 4 of them are ours, and llblgen.com alone pulls 12GB traffic a month simple_smile

<Gulp> flushed Never mind...

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 20:55:06   

Answer wrote:

Honest to god, whenever i have a problem, someone else has a problem, which results in me not being able to fix my problem simple_smile

haha smile , well I wished I could say that here. Obviously I could blame someone, but that doesn't fix the prob. wink (unfortunately. I tried, but it didn't work out wink ).

Jeff: yeah, it's a lot of traffic, I wished it was lower. One of the other sites (thus not our 4 sites), a customer of ours, is also pushing a lot of bytes each day. The current problem is our CMS, which is written in windows DNA: Stored procs, VB COM, XML and asp. So moving a customer off the box to a virtual dir on a big win2k3 box using shared hosting is impossible disappointed . Porting all the stuff to .NET is also not that great. I wrote a reader for the CMS data for viewing in ASP.NET using llblgen pro for the llblgen website, but the editors at the back are still asp 3.0 stuck_out_tongue_winking_eye (and it took us a year to write it, so I don't expect to port it to .NET erm... soon wink )

Frans Bouma | Lead developer LLBLGen Pro
jeffreygg
User
Posts: 805
Joined: 26-Oct-2003
# Posted on: 20-Jul-2004 21:25:33   

Not sure why you want to get it onto another box but if you do, I think MS's server virtualization software's due out soon. That is, if you want to trust version 1.0 software for mission critical hosting. simple_smile

Jeff...

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 21:35:57   

As a good MVP, I always distrust v1.0 software coming from redmond wink

Our co-location ISP has big win2k3 boxes where you can host your site (ASP.NET with sqlserver simple_smile ) as well, perhaps we'll do that for our mission critical stuff. However dell has to fix this box nevertheless.

(although it still works now, strange but ok by me! simple_smile )

Frans Bouma | Lead developer LLBLGen Pro
jeffreygg
User
Posts: 805
Joined: 26-Oct-2003
# Posted on: 20-Jul-2004 21:38:49   

Otis wrote:

As a good MVP, I always distrust v1.0 software coming from redmond wink

Hehe, Microsoft's lucky to have you wink

Jeff...

swallace
User
Posts: 648
Joined: 18-Aug-2003
# Posted on: 20-Jul-2004 22:49:34   

yeah, it's a lot of traffic, I wished it was lower.

One thing I did to reduce traffic was to carefully manage ViewState in ASP.NET. I took a look at your home page, roughly 75k file, and 22k of it was ViewState (probably from the grid control).

If you carefully turn off the use of ViewState for controls on your form you'll find you save a great deal.

Of course, that doesn't help with downloads of demo products and other binaries, but every little bit can help.

swallace
User
Posts: 648
Joined: 18-Aug-2003
# Posted on: 20-Jul-2004 22:51:23   

Following up on my own post:

Here's a great article:

Understanding ASP.NET View State Scott Mitchell 4GuysFromRolla.com

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnaspp/html/viewstate.asp

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 20-Jul-2004 23:27:07   

thanks Scott! simple_smile

The traffic for the site is mostly caused by demo downloads and customers downloading upgrades. A demo is now 6.2MB, which can become quite a big load in a month time, but it's ok simple_smile Just not 'local hosting material' wink

Frans Bouma | Lead developer LLBLGen Pro
wayne avatar
wayne
User
Posts: 611
Joined: 07-Apr-2004
# Posted on: 21-Jul-2004 09:32:48   

The server hosts 12 sites, 4 of them are ours, and llblgen.com alone pulls 12GB traffic a month

12 GB! frowning frowning - What type of line is that server on?

I hope you don't pay for traffic! otherwise it could turn out to be very expensive...

Talking about VIEWSTATE - you might want to look at the flesk control that i discussed with someone in the general section.wink

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 21-Jul-2004 09:46:54   

wayne wrote:

The server hosts 12 sites, 4 of them are ours, and llblgen.com alone pulls 12GB traffic a month

12 GB! frowning frowning - What type of line is that server on?

It's in a 19" rack in a datacenter very near the dutch internet connection point simple_smile The box is on a 100MBit line.

I hope you don't pay for traffic! otherwise it could turn out to be very expensive...

Pay is per GB, but it's not that much. The 12GB was in June when we had a big spike in traffic due to the release of the new version, which was 2MB in size bigger than its predecessor simple_smile .

Talking about VIEWSTATE - you might want to look at the flesk control that i discussed with someone in the general section.wink

I already use HTTP compression and all viewstates are disabled (or very small). Most traffic is from downloads. But keeping an eye on the viewstate is indeed key. If possible use repeaters, not datagrids on a webpage, as a datagrid has a huge viewstate.

Frans Bouma | Lead developer LLBLGen Pro
wayne avatar
wayne
User
Posts: 611
Joined: 07-Apr-2004
# Posted on: 21-Jul-2004 10:04:12   

Pay is per GB, but it's not that much. The 12GB was in June when we had a big spike in traffic due to the release of the new version, which was 2MB in size bigger than its predecessor .

How much do you pay per GB?

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 21-Jul-2004 10:16:26   

depends, mostly < 4 euro.

Frans Bouma | Lead developer LLBLGen Pro
wayne avatar
wayne
User
Posts: 611
Joined: 07-Apr-2004
# Posted on: 21-Jul-2004 10:45:27   

Otis wrote:

depends, mostly < 4 euro.

That is very cheapsimple_smile

I pay a fixed monthly rate for hosting of the server - R 1000 (135.50 Euro's). No additional charges for traffic used. R 1000 is very cheap here, other places charge R 2500 and then you have a 5GB Cap limit.

The server has a 2 X 1GB Nic in it and the server shares a 2MB per second line with 3 other server.

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 21-Jul-2004 12:21:43   

wayne wrote:

Otis wrote:

depends, mostly < 4 euro.

That is very cheapsimple_smile

I pay a fixed monthly rate for hosting of the server - R 1000 (135.50 Euro's). No additional charges for traffic used. R 1000 is very cheap here, other places charge R 2500 and then you have a 5GB Cap limit.

Here you can buy a range of traffic. the more you buy the cheaper it is per GB simple_smile

The server has a 2 X 1GB Nic in it and the server shares a 2MB per second line with 3 other server.

This one also has 2 1GB nics, 1 is used, and connected to a 100MBit switch simple_smile . It's pretty nice simple_smile

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 21-Jul-2004 12:39:27   

It was most likely a faulty report from the backplane temp monitor (it is likely it reported a temp which was too high and the system shut down, however the temp wasn't too high). Bios is updated, and we're keeping an eye on it. Dell is looking further to check if something else is wrong.

The system is monitored 24/7 so if something goes wrong, we'll know it. That's it for now, sorry for the inconvenience it might have caused.

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39776
Joined: 17-Aug-2003
# Posted on: 24-Jul-2004 10:05:16   

Well, that wasn't it. This morning again the server fell flat on its face. Logs show it wasn't the temperature measurement, so it's something else. I hope the sucker stays up this weekend, as support of Dell is even slower in the weekend. rage

Sorry for this inconvenience.

Frans Bouma | Lead developer LLBLGen Pro
wayne avatar
wayne
User
Posts: 611
Joined: 07-Apr-2004
# Posted on: 26-Jul-2004 09:34:37   

Otis wrote:

Well, that wasn't it. This morning again the server fell flat on its face. Logs show it wasn't the temperature measurement, so it's something else. I hope the sucker stays up this weekend, as support of Dell is even slower in the weekend. rage

Sorry for this inconvenience.

Thats what you get when you use Dellwink stuck_out_tongue_winking_eye Nice Black boxes with HUGE price tags.

Do a RAM check - could be faulty RAM. I had a problem about 6months ago with my box - The web application gave these weird error messages sometimes - but worked fine most of the time - it turned out to be a bad 1GB RAM stick.

1  /  2