Results 1 to 11 of 11

Thread: Explanation of tonight server outage, and two other brief outages this week

  1. #1
    Join Date
    Aug 2007
    Posts
    67,743

    Default Explanation of tonight server outage, and two other brief outages this week

    Ok, I'm typing this while _____ angry, so please excuse any typos or other nonsensical details.

    Twice in the last seven days the server was unreachable for about 20 minutes. In both of these cases it was a network problem at the data center where the server is housed. They have been having some network accounts on a few servers that have been impacted the whole network until they can mitigate the attacks.

    Tonight, I got a text alert, so I assumed it was the same thing, as I was told they were doing some configuration changes, but I could see some more of those outages over the next week.

    Both my monitoring and the Datacenters monitoring alerted on BF being down, but unfortunately, they didn't take immediate action, even though it is under management with them and they are supposed to. I was in the shower when I got the text message, so it had been offline for about 20 minutes before I knew about it and contacted support, assuming it was another network problem.

    After around 30 minutes, the server still wasn't up and I start investigating myself. Earlier in the day, the datacenter was supposed to be running a cable between this server, and a small ATOM server I keep to store backups, in case of a major failure of this server. This cable was hooked to the secondary LAN adapter and they had to setup a private IP address so the two servers could talk to each other and the backups could be transferred over that cable and not impact the site performance each morning when the backups are copied to the backup server.

    Anyway, I had never gotten an update on that ticket, so I assumed they put it off until tomorrow, but when this happened, I figured they might have misconfigured the network. I had given VERY specific instructions to check and make sure BroncosForums.com was loading after they configured that secondary LAN adapter, but obviously that wasn't done.

    Bottom line, a tech screwed up the configuration of the secondary LAN (more accurately, I believe it was how he restarted the network service on the server), but more importantly, didn't catch it because he failed to follow my instructions to check that BF was loading properly.

    Until I pointed out that this was probably what happened, they didn't discover their error, fix it and get BF, but not after it was down for about 50 minutes.

    This was a major screw up on the data center's part. They have typically been very reliable and while I am pissed beyond words right now, at this point I'm going to assume this is an anomaly and hopefully we don't suffer this kind of screw up again.

    Sorry about the downtime.

    T

  2. The Following 4 Users High Fived Tned For This Post:


  3. #2
    Join Date
    Aug 2007
    Location
    Westminster, CO
    Adopted Bronco:
    Phillip, Demaryius, Derek, Shane, Von,
    Posts
    47,828

    Default

    Tned -Thanks so much for all you do for Broncos Forums. Is this something that they should be reimbursing you for?

    Thanks to MasterShake for my great signature
    Rest in Peace - Demaryius (88) - Darrent (27) - Damien (29) - Kenny (11)
    #7 - JOHN - #44 - FLOYD - #80 - ROD
    THIS ONES FOR JOHN
    WOULD YOU RATHER WIN UGLY, OR LOSE PRETTY?

  4. The Following User High Fived Denver Native (Carol) For This Post:


  5. #3
    Join Date
    Feb 2009
    Adopted Bronco:
    Demaryius Thomas
    Posts
    31,262

    Default

    No worries man. Still the best site on the internet.
    Let's Rid3!!!!

  6. The Following 2 Users High Fived chazoe60 For This Post:


  7. #4
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by Denver Native (Carol) View Post
    Tned -Thanks so much for all you do for Broncos Forums. Is this something that they should be reimbursing you for?
    The GM will likely offer me a credit of some kind as this is a major bleepity, bleep on their part which I had to point out to them or it would likely still be down. Fortunately, a guy in my IT department at work made the same mistake on a new webserver we put in place (we are a Windows based company and this is really our first Linux/Cpanel server).

    Anyway, I'm rambling because I'm fuming.

  8. The Following User High Fived Tned For This Post:


  9. #5
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    And, they did the same thing again, but this time it was only offline for 10 minutes. I'm betting they made the same mistake. Grrrr

  10. #6
    Join Date
    Aug 2007
    Location
    Westminster, CO
    Adopted Bronco:
    Phillip, Demaryius, Derek, Shane, Von,
    Posts
    47,828

    Default

    Quote Originally Posted by Tned View Post
    The GM will likely offer me a credit of some kind as this is a major bleepity, bleep on their part which I had to point out to them or it would likely still be down. Fortunately, a guy in my IT department at work made the same mistake on a new webserver we put in place (we are a Windows based company and this is really our first Linux/Cpanel server).

    Anyway, I'm rambling because I'm fuming.
    Maybe you should demand a refund, rather than wait and see if they offer one.

    Thanks to MasterShake for my great signature
    Rest in Peace - Demaryius (88) - Darrent (27) - Damien (29) - Kenny (11)
    #7 - JOHN - #44 - FLOYD - #80 - ROD
    THIS ONES FOR JOHN
    WOULD YOU RATHER WIN UGLY, OR LOSE PRETTY?

  11. #7
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by Denver Native (Carol) View Post
    Maybe you should demand a refund, rather than wait and see if they offer one.
    No, overall this company has been very good and the GM very accommodating to various needs I have had. My bigger issue than any credit is an investigation and action on their part to avoid this type of mistake causing future downtime.

  12. The Following User High Fived Tned For This Post:


  13. #8
    Join Date
    Nov 2007
    Location
    Sheridan, Wyoming
    Adopted Bronco:
    Patrick Surtain II
    Posts
    6,389

    Default

    Quote Originally Posted by Tned View Post
    No, overall this company has been very good and the GM very accommodating to various needs I have had. My bigger issue than any credit is an investigation and action on their part to avoid this type of mistake causing future downtime.
    That sounds like the most sensible course of action.
    2009 & 2020 Broncos Forums Pick'em King
    2011 & 2019 Broncos Forums Survival Football Champion


  14. #9

    Default

    I think you should kick somebody in the family jewels.


  15. #10
    Join Date
    Aug 2007
    Location
    Westminster, CO
    Adopted Bronco:
    Phillip, Demaryius, Derek, Shane, Von,
    Posts
    47,828

    Default

    Quote Originally Posted by Tned View Post
    No, overall this company has been very good and the GM very accommodating to various needs I have had. My bigger issue than any credit is an investigation and action on their part to avoid this type of mistake causing future downtime.
    WELL - they sure have not been very good about getting the casino back - LOL

    Thanks to MasterShake for my great signature
    Rest in Peace - Demaryius (88) - Darrent (27) - Damien (29) - Kenny (11)
    #7 - JOHN - #44 - FLOYD - #80 - ROD
    THIS ONES FOR JOHN
    WOULD YOU RATHER WIN UGLY, OR LOSE PRETTY?

  16. #11
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by Denver Native (Carol) View Post
    WELL - they sure have not been very good about getting the casino back - LOL
    lol, you have to blame vBulletin and the author of that add on. That's not the server company.

  17. The Following User High Fived Tned For This Post:


Go
Shop AFC Champions and Super Bowl gear at the official online Pro Shop of the Denver Broncos!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 3
    Last Post: 03-15-2012, 10:26 AM
  2. Another server outage tonight
    By Tned in forum Broncos Talk
    Replies: 27
    Last Post: 01-01-2011, 04:20 PM
  3. Explanation of server downtime tonight
    By Tned in forum Broncos Talk
    Replies: 4
    Last Post: 12-29-2010, 03:19 AM
  4. SERVER MAINTAINANCE NOTICE: Could be brief outages
    By Tned in forum Town Hall Discussion
    Replies: 6
    Last Post: 01-28-2009, 01:03 AM
  5. IMPORTANT: Doing some server tweaking - brief outages could occur
    By TnedTest in forum Town Hall Discussion
    Replies: 17
    Last Post: 11-28-2008, 10:08 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
status.broncosforums.com - BroncosForums status updates
Partner with the USA Today Sports Media Group