Thread: BroncosForums server will be offline for 1-3 hours tomorrow (FRI) morning - check http://status.broncosforums.com/ for updates

I don't like doing this so close to the season, but I really don't want to have to do it during the season.

Tomorrow morning, after all the backups run and are transferred off the server (6:00 am EDT) the server will go offline to have some disk maintenance performed. If all goes well this process will probably take 60-90 minutes. If there are any issues, it could take several hours. If the process fails, it will require reloading the server, and reloading BF from a backup, which could take anywhere from 3-6 hours.

All of that said, the most likely downtime is probably 60-90 minutes unless a major problem occurs.

For those techies that are curious about the details (talking about you Thnik)...

When this new server was brought online in April, two things were done that weren't ideal. First, the /tmp directory was created too small considering the size of the BroncosForums database. It probably should have been twice the size it was actually created. Second, it was not placed on the primary, fast SSD drive array, but on the slower traditional SATA hard drive array that was intended to only house backups.

The issue that needs to be addressed is the size. With growing frequency, some vBulletin queries have been failing due to the /tmp partition filling up. While it is still only happening once or twice a day, the frequency has been increasing. As the size of the database will continue to grow, this problem will get worse all season long. So, it's necessary to bite the bullet and fix it now. This requires re partitioning the primary hard drive, to free up space for the larger /tmp partition.

One benefit of doing this is I will have the /tmp directory moved to the SSD array, which will provide better performance when the database uses /tmp for large queries, searches and stuff.

Anyway. Hopefully, by 7:00 or 8:00 AM eastern time tomorrow, everything will be back online and running fine.

If there is an extended outage, I will post updates at http://status.broncosforums.com/ and on Twitter (follow @BroncosForums)

Also, if you just noticed a momentary freeze, or there is sluggishness over the next 20 minutes, for some belt and suspenders insurance, I am transferring a copy of BroncosForums over to the virtual server where status.broncosForums.com resides. While vastly under-powered to run BF, it could serve as a temporary emergency home if necessary. I will make sure all the configurations and tweaks are made to that server to run BF for a short sting if things go really wrong tomorrow.

As always TNed, you're the man. Thanks for giving us all an internet home.

This is bullshit! I'm outta here!

Originally Posted by chazoe60

This is bullshit! I'm outta here!

Yea, yea, if I wasn't such a tea-totaling, southern boy that never cursed I would tell you ________________________.

Ok, as a worst case scenario, I have a copy of the site as it was at 9:00 tonight (CDT) running on the virtual server, and I upgraded to the largest plan they offer. If things go totally pear shaped (as our Brit friends would say), I'll change the DNS records to point to that -- hopefully with a current backup, but in worst case as the server was at 9:00.

I do think that any serious problems are pretty unlikely, and the partition resize typically works with no problems, but there are no guarantees.

Bad news is the resizing of the /tmp partition failed. The good news, is that it was early enough in the process that the server could be brought back online as it was before. So, we still have the issue that will continue to happen, probably with increasing frequency.

I'm looking at my other options now. May possibly try some other options over the weekend. I'll post an update if the server will be going down again.

a lack of planning on your part and we have to suffer the consequences? this is bs Tned

Shut your mouth Nut and apologize to tned right this instant

Thank you Tned for doing everything that you do. I was just worried that it would happened 10 seconds before the cuts started to come out and I would be screaming NOOOOOO. Great job as always....

I want my money back or I will be contacting Tom martino

Trying a MySQL (vs. Linux level) workaround to the temp directory issue. It shouldn't result in any downtime (maybe a momentary blip) and then I'll keep an eye on things in the coming weeks and hopefully it resolves the problem completely, or at least buys enough time to deal with it fully after the season.

Thanks for your hard work, Tned!


As far as Nut, get him some lotion and a nudie magazine and he'll be fine!