Page 1 of 2 1 2 LastLast
Results 1 to 15 of 16

Thread: 24+ server outage from 9/7 to 9/8

  1. #1
    Join Date
    Aug 2007
    Posts
    67,743

    Default 24+ server outage from 9/7 to 9/8

    UPDATE 9/9/13: See original post below.

    Ok, I know a little more about what happened. A relatively recent security problem with vBulletin was discovered, where if the install directory was not deleted after an install/upgrade, it allowed a hacker to create an administrator account. For years, vBulletin didn't recommend deleting this directory, but instead only the install.php file. That changed recently. A few weeks ago they posted a notice about a possible exploit on their support site telling people to delete their installs directory, but didn't send an email. Then, apparently last week, a TON of sites got hacked all at once, so they sent an email on 9/3 with the same information that had been on their support site.

    I was tied up with work, and still recovering from my house flood, so not keeping up with email. So, I didn't see it until days later. As it turns out, they had created an admin account on the 3rd on BF, so even if I had read the email and deleted the install directory right away, it would have been too late as they already had access, because in one fell swoop they compromised hundreds or thousands of vBulletin servers, then days later, started messing with them.

    I have been working with the server company, my old support company that I'm not under contract with anymore but paying hourly, and vB support, and we believe we have covered all the bases, but vBulletin is being pretty tight lipped about all the details, as they try and determine the best way to avoid it in the future (besides just deleting the install directory).

    I'm also considering changing server companies, even though I've been with this one for five years. While the hack had nothing to do with them, their response time and some other issues were unacceptable. If I decide to do it, I will probably do it next Monday, and the actual downtime should be minimal, these days DNS usually updates quickly, and it should only be 30 minutes to a few hours. My work schedule is nasty now, and then I have one week where it's not to bad, then I'm traveling part of the week of the 23rd, week of the 6th, and then out of the country for 15 days starting 10/15, so I need to have datacenter that will react quickly if we have another problem.

    I'll post another update in a day or two if I learn more about what happened, or decide to move ahead with the datacenter change.


    =========================
    Ok, what happened?


    The server was hacked yesterday. In order to be 100% careful, I immediately took BroncosForums down, while the support people investigated what happened. Once it was determined that the server was definitely compromised, then the decision was made to rebuild the server from scratch.


    First, we had to have a gameplan for rolling all files back to a known good point, but not losing and data/posts. Then, it was a very slow process to clone the old drives for a backup, reinstall, reload, etc. I grabbed a couple hours sleep and got up at 3:00 am, and worked with the support people to finish, but it still took them until nearly 9:00 before they had finished their part of the server reload, security hardening, etc.


    I then started the backup restore, and then we had some major problems getting that to work on the newly installed server (you probably saw the pages very messed up when you first saw the server come back up). That took many hours to figure out.


    Anyway, we are now on a freshly reloaded server, with the latest vBulletin and other software.


    I’ll provide further details and answer any questions later, but now I have to travel about 1 hour to a work training that I’m very late for. It will go until midnight or so, but I’ll be keeping my eye on things via my phone.
    Last edited by Tned; 09-09-2013 at 10:49 PM.


  2. #2
    Join Date
    Aug 2007
    Location
    New Orleans, LA
    Adopted Bronco:
    DT
    Posts
    41,696

    Default

    Thanks for all the hard work Tned! If there is any silver lining in all of this it's that this happened last night instead of Thursday!

  3. The Following 7 Users High Fived Davii For This Post:


  4. #3
    Join Date
    Aug 2007
    Location
    Tucson, AZ
    Adopted Bronco:
    Josey Jewell
    Posts
    30,216

    Default

    Just a further testament to the fact that this board would not ever have made it without you, T. Name:  thdrink.gif
Views: 145
Size:  381 Bytes
    Though He slay me, I will trust in Him . . . (Job 13:15)


  5. The Following 7 Users High Fived topscribe For This Post:


  6. #4
    Join Date
    Aug 2007
    Location
    Sharkdom of Snowdonia
    Adopted Bronco:
    All of them
    Posts
    4,506

    Default

    It was a rattled ratbird seeking revenge most likely who hacked.
    What a lot of work for you.
    Glad you've been able to sort things out.
    Crossing everything there aren't anymore troubles.
    Thanks Tned.

  7. The Following 5 Users High Fived Jaws For This Post:


  8. #5
    Join Date
    Aug 2007
    Posts
    37,291

    Default

    T, thank you for spending your weekend on the shit storm to give us this place we love.

  9. The Following 12 Users High Fived GEM For This Post:


  10. #6

    Default

    Thanks so much for giving us all this fun and pleasure.... !!!!


  11. The Following User High Fived Joker56 For This Post:


  12. #7
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    I also updated the first post in this thread with this update:

    UPDATE 9/9/13: See original post below.

    Ok, I know a little more about what happened. A relatively recent security problem with vBulletin was discovered, where if the install directory was not deleted after an install/upgrade, it allowed a hacker to create an administrator account. For years, vBulletin didn't recommend deleting this directory, but instead only the install.php file. That changed recently. A few weeks ago they posted a notice about a possible exploit on their support site telling people to delete their installs directory, but didn't send an email. Then, apparently last week, a TON of sites got hacked all at once, so they sent an email on 9/3 with the same information that had been on their support site.

    I was tied up with work, and still recovering from my house flood, so not keeping up with email. So, I didn't see it until days later. As it turns out, they had created an admin account on the 3rd on BF, so even if I had read the email and deleted the install directory right away, it would have been too late as they already had access, because in one fell swoop they compromised hundreds or thousands of vBulletin servers, then days later, started messing with them.

    I have been working with the server company, my old support company that I'm not under contract with anymore but paying hourly, and vB support, and we believe we have covered all the bases, but vBulletin is being pretty tight lipped about all the details, as they try and determine the best way to avoid it in the future (besides just deleting the install directory).

    I'm also considering changing server companies, even though I've been with this one for five years. While the hack had nothing to do with them, their response time and some other issues were unacceptable. If I decide to do it, I will probably do it next Monday, and the actual downtime should be minimal, these days DNS usually updates quickly, and it should only be 30 minutes to a few hours. My work schedule is nasty now, and then I have one week where it's not to bad, then I'm traveling part of the week of the 23rd, week of the 6th, and then out of the country for 15 days starting 10/15, so I need to have datacenter that will react quickly if we have another problem.

    I'll post another update in a day or two if I learn more about what happened, or decide to move ahead with the datacenter change.

  13. The Following 6 Users High Fived Tned For This Post:


  14. #8
    Join Date
    Jan 2009
    Location
    Phoenix, AZ
    Adopted Bronco:
    Phillip "TD" Lindsay
    Posts
    11,300

    Default

    I've been getting "Unexpected Server Response" messages on about 2/3 of the threads that I've tried to access with my Android tablet.
    I’m an Autistic Self-Advocate. If you have any questions about Autism/Asperger’s, feel free to ask. I’m not offended by any question asked by anyone who has a genuine desire to understand us better.

    https://aacphoenix.com/

  15. #9
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Using a mobile app or browser? If browser, which one?

  16. #10
    Join Date
    Sep 2007
    Location
    colorado
    Posts
    26,922

    Default

    So what exactly could/would they do? The hackers I mean. Just change people's names and posts and add and delete stuff and such? Steal their emails and sell them and ip trace and such?


    I thoroughly enjoy messing with people on the internets but that seems like a lot of work for little entertainment.
    The Plan at the moment:

    Draft: Trade a 3rd and 6th this year to a team to move up and get a 2nd next year (this will happen).

    Players I want:
    Jake Ferguson (Jake Butt) or Jelani Woods or Jeremy Ruckert or Cade Otten (owen daniels) at TE- All 4th rd or later.
    Troy Anderson LB 3rd/4th rd (yay Timmy!)
    Neil Farrell, JR DL- run stuffer- bye purcell

  17. #11
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by underrated29 View Post
    So what exactly could/would they do? The hackers I mean. Just change people's names and posts and add and delete stuff and such? Steal their emails and sell them and ip trace and such?


    I thoroughly enjoy messing with people on the internets but that seems like a lot of work for little entertainment.
    It appears to vary greatly by site that was hacked. Some sites, they simply modified the homepage to say something like, "delete your install folder or you will get hacked again" and in other cases, they redirected all traffic to a Mirynmar Liberation Movement (sp??), others they replaced the sites ads with their own. In this site, they don't appear to have done anything with vBulletin or BroncosForums, other than using it as a means to launch a script that allowed them to create a root level, super user account on the server itself. Once we received alerts about a new superuser being created, we started taking action, including taking BroncosForums offline, even though the server remained up as it was being investigated.

    My guess, and it's only a guess, is that they were looking through the other hosting accounts on the server (I have some with my personal email, some other sites I play around with, test accounts/sites, etc.) to see if there was anything like CC's or other details, or possibly trying to add the server into part of a botnet. It's hard to tell, because they immediately disabled server logging.

    Because we didn't know what they did, was the reason we felt the only option was to reload the server from scratch.

  18. The Following 3 Users High Fived Tned For This Post:


  19. #12
    Join Date
    Jan 2009
    Location
    Phoenix, AZ
    Adopted Bronco:
    Phillip "TD" Lindsay
    Posts
    11,300

    Default

    Mobile app
    I’m an Autistic Self-Advocate. If you have any questions about Autism/Asperger’s, feel free to ask. I’m not offended by any question asked by anyone who has a genuine desire to understand us better.

    https://aacphoenix.com/

  20. #13
    Join Date
    Aug 2007
    Posts
    67,743

    Default

    Quote Originally Posted by FanInAZ View Post
    Mobile app
    Official Broncosforums or Forum Runner. To be honest, I haven't been using the official. It's possible that my upgrading the server to 4.21 and not upgrading the mobile app (I stopped paying the $99 a year, as they went well over a year without an update), might be causing the problem. Let me do some research on this.

    In the meantime, I highly recommend Forum Runner. Granted, it costs $2.99 (doesn't go to me, goes to Forum Runner), but at the moment it's far superior to the vBulletin mobile app.

  21. The Following 2 Users High Fived Tned For This Post:


  22. #14
    Join Date
    Nov 2011
    Location
    Vernal, Utah
    Posts
    6,592

    Default

    For a long time I couldn't bring myself to buy forum runner when I could use the official app for free. A couple months ago I finally purchased forum runner and it's the best three bucks I've spent
    Free Hotcarl!

  23. The Following User High Fived Pudge For This Post:


  24. #15
    Join Date
    Aug 2007
    Posts
    37,291

    Default

    VBulletin's mobile app sucks. I just log into the regular site and it's pretty easy. Will probably upgrade to Forum Runner.

  25. The Following User High Fived GEM For This Post:


Go
Shop AFC Champions and Super Bowl gear at the official online Pro Shop of the Denver Broncos!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 10
    Last Post: 06-19-2012, 01:02 PM
  2. Another server outage tonight
    By Tned in forum Broncos Talk
    Replies: 27
    Last Post: 01-01-2011, 04:20 PM
  3. Server outage was caused by network problem at datacenter
    By Tned-Mobile in forum Broncos Talk
    Replies: 13
    Last Post: 12-01-2009, 02:40 AM
  4. Server outage this afternoon
    By Tned in forum Town Hall Discussion
    Replies: 27
    Last Post: 05-26-2009, 07:27 PM
  5. Server outage this afternoon
    By Tned in forum Broncos Talk
    Replies: 0
    Last Post: 05-21-2009, 07:18 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
status.broncosforums.com - BroncosForums status updates
Partner with the USA Today Sports Media Group