Page 1 of 2 1 2 LastLast
Results 1 to 15 of 30

Thread: Server downtime this afternoon

  1. #1
    Join Date
    Aug 2007
    Posts
    67,743

    Default Server downtime this afternoon

    Well, we had about an hour of downtime this afternoon. Here's an update about what happened. (beware, there is some tech talk here).

    First, the main reason for the delay in resolving the outage was that I was at a wedding (not mine) out in the boonies with cell reception bouncing between no service and 1 bar. I pay for a monitoring service that checks every 2 minutes to see if Broncosforums.com is up and running, and if it fails two checks in a row, then it sends me a text message/page. So, typically, I am going to be alerted within 4-6 minutes of the server going down. However, because I had limited cell coverage, that automated alert (and an email from Jrwiz) didn't arrive for about 30 minutes after it went down. Typically, this isn't the case and I am alerted almost immediately.

    As to the outage itself, I run Broncosforums.com on what is called Virtual Server. Anyone that follows the tech world or stocks might have heard of Microsoft virtual server or VMware virtual servers. It is the movement in server technology. How it works is you take more powerful machines than you would normally dedicate to a single server, and than use a virtualization software to create multiple 'virtual servers', each one running its own operating system, having dedicated ram and dedicated CPU or processing power.

    Anyway, the amount of money I spend on Broncosforums.com gives me two options. I low end dedicated server or a high-end Virtual Server. I have opted to go with a virtual server for two reasons.

    First, the low end dedicated servers are typically single or possibly dual core processors, a minimal amount of RAM and only one hard drive, or possibly a mirrored hard drive setup for some redundancy in case a hard drive failed. With the virtual server I purchase, it is housed on a high end Dell server with an 8 core (dual - quad core) processor(s), a large Raid 10 disk array with 15,000 RPM drives, 32 gigs of Ram, etc.

    Since I elected to purchase the highest level package they offered, I am guaranteed a large amount of that processing power and a dedicated amount of RAM. The actual performance of a high end virtual server on the high end hardware is much greater performance than a low-end dedicated server.

    Second, with the high end VPS I get more support than if I purchased a dedicated server. If I leased a low-end dedicated server, it would be what they call unmanaged, which means I would have to do everything myself. With the high-end, managed, virtual servers I am expected to do a lot of the server maintainance, upgrades, etc., but I have support staff available to help me if I run into a problem.

    So, what happened? Well, the currently virtualization software that are used in linux hosting can guarantee RAM and CPU time to each virtual server, but does not guarantee disk access. Typically, this isn't a problem, especially on a server with a very fast, RAID-10 disk array like the one Broncosforums.com is hosted on. However, one of the other virtual servers on the hardware node (underlying server) was hacked and apparently completely overloaded the server with disk I/O access. Therefore, other virtual servers, like Broncosforums.com, were essentially locked out from the disk drives, which resulted in database errors.

    Since I upgraded from standard hosting to a virtual server 5 months ago, this is the first time this has happened. We have had an doutage (once brief and once about 30-40 minutes, for other reasons) based on another virtual server being hacked.

    So, at this point, I think the virtual server still offers the best performance for the money. However, if this type of problem becomes more than a very rare occurence, I will consider moving broncosforums.com to a dedicated server. Over the last 5 months, I have learned a great deal about managing a linux server and while I would prefer not to be 'on my own' when something went wrong, I think I could properly mange the server.

    Anyway, if you made it this far in the post, you know the details of the great Broncosforums.com outage of March, 2008. Hopefully, these types of outages are few and far between, and that next time one occurs, I will be in decent cell coverage and get the server down alert much sooner.
    Last edited by Tned; 03-30-2008 at 03:59 PM.

  2. The Following 10 Users High Fived Tned For This Post:


  3. #2
    Join Date
    Dec 2007
    Location
    Elpaso TX
    Adopted Bronco:
    Josh McDaniels Sucks
    Posts
    31,226

    Default

    Thanks Tned for caring so much. The Time, money etc........ its unreal. Thanks again buddy.
    Thanks MO for the wicked Sig.

  4. The Following 2 Users High Fived claymore For This Post:


  5. #3
    Join Date
    Aug 2007
    Location
    Cheshire, England
    Adopted Bronco:
    Champ Bailey
    Posts
    5,703

    Default

    At least it wasn't just me then............I've been paranoid about any failure messages since my internet died!!!

    Appreciate all the work
    BFC.........D Will #27.............D Nash #29..............K Mc #11
    You'll Never Walk Alone



    The Whiskeyteers ~ One for all, and all for more

  6. #4
    Join Date
    Dec 2007
    Adopted Bronco:
    Pat Bowlen
    Posts
    97,306

    Default

    I have no clue what you said, but thank you...now sit down and have a cold one
    *The statements above are my opinions, unless they are links, because then they are links, which wouldn't make them my opinions, and I suppose stats aren't necessarily opinion, but they are certainly presented to support an opinion. Proceed accordingly.

    Quote Originally Posted by Buff View Post
    What is this, amateur hour? It's TNF against the Jets and you didn't think you'd need extra booze?

  7. The Following User High Fived MOtorboat For This Post:


  8. #5
    Join Date
    Dec 2007
    Adopted Bronco:
    Richard Simmons
    Posts
    30,177

    Default

    This is when I am glad the Freak imploded. What a great man. Tned, I you.

  9. The Following User High Fived BeefStew25 For This Post:


  10. #6
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Thanks for the thanks guys and sorry for the downtime.

  11. #7
    Join Date
    Aug 2007
    Posts
    31,747

    Default

    Next time you can't leave town or go somewhere where you have not cell service..

    BTW my email was within a few minutes of it going down as I was trying to reply to a post..

    I rebooted got the same response and emailed..

    Anyway thanks for all you do for us..wayward kids..

  12. #8
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by Jrwiz View Post
    Next time you can't leave town or go somewhere where you have not cell service..

    BTW my email was within a few minutes of it going down as I was trying to reply to a post..

    I rebooted got the same response and emailed..

    Anyway thanks for all you do for us..wayward kids..
    I have the monitoring service I pay for not page me unless it has two consecutive failures, to try and reduce middle of the night false positives (such as if there is just a temp internet hiccup that causes a single failure). It checks every two minutes, so theoretically, the page could go out 5 minutes and 59 seconds after the server goes down, if the server went down the second after the last check. Then, typically the text message/page arrives almost instantaneously, but didn't this time because of the cell reception.

    When they arrived, you had sent your email a couple minutes after the page was sent from the monitoring company, but it took close to 30 minutes before they hit my phone.

  13. #9
    Join Date
    Aug 2007
    Posts
    31,747

    Default

    Quote Originally Posted by Tned View Post
    I have the monitoring service I pay for not page me unless it has two consecutive failures, to try and reduce middle of the night false positives (such as if there is just a temp internet hiccup that causes a single failure). It checks every two minutes, so theoretically, the page could go out 5 minutes and 59 seconds after the server goes down, if the server went down the second after the last check. Then, typically the text message/page arrives almost instantaneously, but didn't this time because of the cell reception.

    When they arrived, you had sent your email a couple minutes after the page was sent from the monitoring company, but it took close to 30 minutes before they hit my phone.
    While cell phones are great we have come to rely on the damned things to much..

    There have been many times it get 3 voice mails all at the same time from a client that has been leaving them over the past 3 hours. Does not happen often, only when they count the most.. And they do not even show up as a missed call..


    Going to church see you later

  14. #10
    Join Date
    Aug 2007
    Location
    Just north of the Mexican Border
    Posts
    1,507

    Default

    Quote Originally Posted by Tned View Post
    Anyway, if you made it this far in the post, you know the details of the great Broncosforums.com outage of March, 2009. Hopefully, these types of outages are few and far between, and that next time one occurs, I will be in decent cell coverage and get the server down alert much sooner.
    slow down speedy..........we are only in 2008

  15. The Following 3 Users High Fived Escobar For This Post:


  16. #11
    Join Date
    Aug 2007
    Location
    Tucson, AZ
    Adopted Bronco:
    Josey Jewell
    Posts
    30,216

    Default

    Quote Originally Posted by Den21vsBal19 View Post
    At least it wasn't just me then............I've been paranoid about any failure messages since my internet died!!!

    Appreciate all the work
    Wow, after all the troubles you've been through, I'll bet it did cause some angst.

    -----
    Though He slay me, I will trust in Him . . . (Job 13:15)


  17. #12
    Join Date
    Aug 2007
    Location
    Cheshire, England
    Adopted Bronco:
    Champ Bailey
    Posts
    5,703

    Default

    Quote Originally Posted by Escobar View Post
    slow down speedy..........we are only in 2008


    Quote Originally Posted by topscribe View Post
    Wow, after all the troubles you've been through, I'll bet it did cause some angst.

    -----
    You'd better believe it!!!!


    Tned's lucky with the text service though, I was Anfield a while back for a Liverpool game, and got a text message from my brother, saying that his train was running late..............I'd last taken him the footie 3 months earlier
    BFC.........D Will #27.............D Nash #29..............K Mc #11
    You'll Never Walk Alone



    The Whiskeyteers ~ One for all, and all for more

  18. #13
    Join Date
    Aug 2007
    Posts
    31,747

    Default

    Quote Originally Posted by Den21vsBal19 View Post




    You'd better believe it!!!!


    Tned's lucky with the text service though, I was Anfield a while back for a Liverpool game, and got a text message from my brother, saying that his train was running late..............I'd last taken him the footie 3 months earlier
    Could you translate this into English please? For us Amrcian dummies..

  19. The Following User High Fived Lonestar For This Post:


  20. #14
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by Den21vsBal19 View Post




    You'd better believe it!!!!


    Tned's lucky with the text service though, I was Anfield a while back for a Liverpool game, and got a text message from my brother, saying that his train was running late..............I'd last taken him the footie 3 months earlier
    Quote Originally Posted by Jrwiz View Post
    Could you translate this into English please? For us Amrcian dummies..
    He was in anfield to watch a Liverpool soccer game.

    He received a text message from his brother, stating that his train was running late.

    The text message was sent three months earlier when he last was meeting his brother to go to a soccer game.

    How did I do with the translation, Den?

  21. #15
    Join Date
    Aug 2007
    Posts
    31,747

    Default

    Quote Originally Posted by Tned View Post
    He was in anfield to watch a Liverpool soccer game.

    He received a text message from his brother, stating that his train was running late.

    The text message was sent three months earlier when he last was meeting his brother to go to a soccer game.

    How did I do with the translation, Den?
    Try again some of us missed that class in school

Go
Shop AFC Champions and Super Bowl gear at the official online Pro Shop of the Denver Broncos!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Scheduled server downtime Friday AM
    By Tned in forum Broncos Talk
    Replies: 13
    Last Post: 05-09-2008, 11:24 AM
  2. Scheduled server downtime Friday AM
    By Tned in forum What's on your Mind (Chit Chat)
    Replies: 3
    Last Post: 05-09-2008, 10:58 AM
  3. Replies: 16
    Last Post: 01-21-2008, 09:26 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
status.broncosforums.com - BroncosForums status updates
Partner with the USA Today Sports Media Group