Tned
03-29-2008, 08:41 PM
Well, we had about an hour of downtime this afternoon. Here's an update about what happened. (beware, there is some tech talk here).
First, the main reason for the delay in resolving the outage was that I was at a wedding (not mine) out in the boonies with cell reception bouncing between no service and 1 bar. I pay for a monitoring service that checks every 2 minutes to see if Broncosforums.com is up and running, and if it fails two checks in a row, then it sends me a text message/page. So, typically, I am going to be alerted within 4-6 minutes of the server going down. However, because I had limited cell coverage, that automated alert (and an email from Jrwiz) didn't arrive for about 30 minutes after it went down. Typically, this isn't the case and I am alerted almost immediately.
As to the outage itself, I run Broncosforums.com on what is called Virtual Server. Anyone that follows the tech world or stocks might have heard of Microsoft virtual server or VMware virtual servers. It is the movement in server technology. How it works is you take more powerful machines than you would normally dedicate to a single server, and than use a virtualization software to create multiple 'virtual servers', each one running its own operating system, having dedicated ram and dedicated CPU or processing power.
Anyway, the amount of money I spend on Broncosforums.com gives me two options. I low end dedicated server or a high-end Virtual Server. I have opted to go with a virtual server for two reasons.
First, the low end dedicated servers are typically single or possibly dual core processors, a minimal amount of RAM and only one hard drive, or possibly a mirrored hard drive setup for some redundancy in case a hard drive failed. With the virtual server I purchase, it is housed on a high end Dell server with an 8 core (dual - quad core) processor(s), a large Raid 10 disk array with 15,000 RPM drives, 32 gigs of Ram, etc.
Since I elected to purchase the highest level package they offered, I am guaranteed a large amount of that processing power and a dedicated amount of RAM. The actual performance of a high end virtual server on the high end hardware is much greater performance than a low-end dedicated server.
Second, with the high end VPS I get more support than if I purchased a dedicated server. If I leased a low-end dedicated server, it would be what they call unmanaged, which means I would have to do everything myself. With the high-end, managed, virtual servers I am expected to do a lot of the server maintainance, upgrades, etc., but I have support staff available to help me if I run into a problem.
So, what happened? Well, the currently virtualization software that are used in linux hosting can guarantee RAM and CPU time to each virtual server, but does not guarantee disk access. Typically, this isn't a problem, especially on a server with a very fast, RAID-10 disk array like the one Broncosforums.com is hosted on. However, one of the other virtual servers on the hardware node (underlying server) was hacked and apparently completely overloaded the server with disk I/O access. Therefore, other virtual servers, like Broncosforums.com, were essentially locked out from the disk drives, which resulted in database errors.
Since I upgraded from standard hosting to a virtual server 5 months ago, this is the first time this has happened. We have had an doutage (once brief and once about 30-40 minutes, for other reasons) based on another virtual server being hacked.
So, at this point, I think the virtual server still offers the best performance for the money. However, if this type of problem becomes more than a very rare occurence, I will consider moving broncosforums.com to a dedicated server. Over the last 5 months, I have learned a great deal about managing a linux server and while I would prefer not to be 'on my own' when something went wrong, I think I could properly mange the server.
Anyway, if you made it this far in the post, you know the details of the great Broncosforums.com outage of March, 2008. Hopefully, these types of outages are few and far between, and that next time one occurs, I will be in decent cell coverage and get the server down alert much sooner.
First, the main reason for the delay in resolving the outage was that I was at a wedding (not mine) out in the boonies with cell reception bouncing between no service and 1 bar. I pay for a monitoring service that checks every 2 minutes to see if Broncosforums.com is up and running, and if it fails two checks in a row, then it sends me a text message/page. So, typically, I am going to be alerted within 4-6 minutes of the server going down. However, because I had limited cell coverage, that automated alert (and an email from Jrwiz) didn't arrive for about 30 minutes after it went down. Typically, this isn't the case and I am alerted almost immediately.
As to the outage itself, I run Broncosforums.com on what is called Virtual Server. Anyone that follows the tech world or stocks might have heard of Microsoft virtual server or VMware virtual servers. It is the movement in server technology. How it works is you take more powerful machines than you would normally dedicate to a single server, and than use a virtualization software to create multiple 'virtual servers', each one running its own operating system, having dedicated ram and dedicated CPU or processing power.
Anyway, the amount of money I spend on Broncosforums.com gives me two options. I low end dedicated server or a high-end Virtual Server. I have opted to go with a virtual server for two reasons.
First, the low end dedicated servers are typically single or possibly dual core processors, a minimal amount of RAM and only one hard drive, or possibly a mirrored hard drive setup for some redundancy in case a hard drive failed. With the virtual server I purchase, it is housed on a high end Dell server with an 8 core (dual - quad core) processor(s), a large Raid 10 disk array with 15,000 RPM drives, 32 gigs of Ram, etc.
Since I elected to purchase the highest level package they offered, I am guaranteed a large amount of that processing power and a dedicated amount of RAM. The actual performance of a high end virtual server on the high end hardware is much greater performance than a low-end dedicated server.
Second, with the high end VPS I get more support than if I purchased a dedicated server. If I leased a low-end dedicated server, it would be what they call unmanaged, which means I would have to do everything myself. With the high-end, managed, virtual servers I am expected to do a lot of the server maintainance, upgrades, etc., but I have support staff available to help me if I run into a problem.
So, what happened? Well, the currently virtualization software that are used in linux hosting can guarantee RAM and CPU time to each virtual server, but does not guarantee disk access. Typically, this isn't a problem, especially on a server with a very fast, RAID-10 disk array like the one Broncosforums.com is hosted on. However, one of the other virtual servers on the hardware node (underlying server) was hacked and apparently completely overloaded the server with disk I/O access. Therefore, other virtual servers, like Broncosforums.com, were essentially locked out from the disk drives, which resulted in database errors.
Since I upgraded from standard hosting to a virtual server 5 months ago, this is the first time this has happened. We have had an doutage (once brief and once about 30-40 minutes, for other reasons) based on another virtual server being hacked.
So, at this point, I think the virtual server still offers the best performance for the money. However, if this type of problem becomes more than a very rare occurence, I will consider moving broncosforums.com to a dedicated server. Over the last 5 months, I have learned a great deal about managing a linux server and while I would prefer not to be 'on my own' when something went wrong, I think I could properly mange the server.
Anyway, if you made it this far in the post, you know the details of the great Broncosforums.com outage of March, 2008. Hopefully, these types of outages are few and far between, and that next time one occurs, I will be in decent cell coverage and get the server down alert much sooner.