Tned
08-30-2012, 08:40 PM
I don't like doing this so close to the season, but I really don't want to have to do it during the season.
Tomorrow morning, after all the backups run and are transferred off the server (6:00 am EDT) the server will go offline to have some disk maintenance performed. If all goes well this process will probably take 60-90 minutes. If there are any issues, it could take several hours. If the process fails, it will require reloading the server, and reloading BF from a backup, which could take anywhere from 3-6 hours.
All of that said, the most likely downtime is probably 60-90 minutes unless a major problem occurs.
For those techies that are curious about the details (talking about you Thnik)...
When this new server was brought online in April, two things were done that weren't ideal. First, the /tmp directory was created too small considering the size of the BroncosForums database. It probably should have been twice the size it was actually created. Second, it was not placed on the primary, fast SSD drive array, but on the slower traditional SATA hard drive array that was intended to only house backups.
The issue that needs to be addressed is the size. With growing frequency, some vBulletin queries have been failing due to the /tmp partition filling up. While it is still only happening once or twice a day, the frequency has been increasing. As the size of the database will continue to grow, this problem will get worse all season long. So, it's necessary to bite the bullet and fix it now. This requires re partitioning the primary hard drive, to free up space for the larger /tmp partition.
One benefit of doing this is I will have the /tmp directory moved to the SSD array, which will provide better performance when the database uses /tmp for large queries, searches and stuff.
Anyway. Hopefully, by 7:00 or 8:00 AM eastern time tomorrow, everything will be back online and running fine.
If there is an extended outage, I will post updates at http://status.broncosforums.com/ and on Twitter (follow @BroncosForums)
Tomorrow morning, after all the backups run and are transferred off the server (6:00 am EDT) the server will go offline to have some disk maintenance performed. If all goes well this process will probably take 60-90 minutes. If there are any issues, it could take several hours. If the process fails, it will require reloading the server, and reloading BF from a backup, which could take anywhere from 3-6 hours.
All of that said, the most likely downtime is probably 60-90 minutes unless a major problem occurs.
For those techies that are curious about the details (talking about you Thnik)...
When this new server was brought online in April, two things were done that weren't ideal. First, the /tmp directory was created too small considering the size of the BroncosForums database. It probably should have been twice the size it was actually created. Second, it was not placed on the primary, fast SSD drive array, but on the slower traditional SATA hard drive array that was intended to only house backups.
The issue that needs to be addressed is the size. With growing frequency, some vBulletin queries have been failing due to the /tmp partition filling up. While it is still only happening once or twice a day, the frequency has been increasing. As the size of the database will continue to grow, this problem will get worse all season long. So, it's necessary to bite the bullet and fix it now. This requires re partitioning the primary hard drive, to free up space for the larger /tmp partition.
One benefit of doing this is I will have the /tmp directory moved to the SSD array, which will provide better performance when the database uses /tmp for large queries, searches and stuff.
Anyway. Hopefully, by 7:00 or 8:00 AM eastern time tomorrow, everything will be back online and running fine.
If there is an extended outage, I will post updates at http://status.broncosforums.com/ and on Twitter (follow @BroncosForums)