Problems posting Thursday night
I had put this notice up overnight, but am now taking it down now so people don't have to see it every time they login. Here's what I had posted last night about the database errors that were occurring when trying to create a new post.
Quote:
Sometime Thursday the BroncosForums vBulletin database became corrupt. We are attempting to repair the table, but so far we have had no success.
A further complication is that the database backup performed at 11:15pm Thursday, also appears to be corrupt, so if we are unable to repair the database, we will have to revert back to the backup from 11:15 am Thursday morning.
I will get this resolved as soon as possible. Tned
Update 12:45 AM EDT: it appears the table that was corrupt is a search table. It's what is checked when we search for posts on BF. While multiple repairs have failed, it now says the table is repaired. However, this happened earlier and then it became corrupt again. I may need to delete that table and have vBulletin rebuild the index which will take many hours.
For now, it is ok to post. In fact, it is a good thing, since every new post writes to that table and it will help determine if it is fixed.
Update 1:45 AM EDT: While the errors seemed to go away, the search index did not appear correct. I did some test searches and the results didn't seem right. Therefore, I deleted and recreated the table that was corrupt (to try and insure there is no residual problem), and then deleted all search indexes and started a search index rebuild.
This search rebuild can take a while on 1.5 million posts. It's moving along at a pretty good clip, so if it doesn't slow down, it should finish in 2-3 hours. Until it finishes, most searches will return no threads, or only the ones that it has indexed so far.
Please note that "New Posts" IS a search, so until this reindexing of the search data completes, the new post button (and today's posts) will not be reliable. Please go directly to the forums (Broncos Talk, Other NFL, Lounge, etc.) to check for new posts/threads.
You can remove this notice by clicking the x to the upper right of this notice box.
The rebuilding of the indexes worked successfully, and the best I can tell, everything is back to normal with no data/posts lost. I'm not sure exactly how long the problem was happening yesterday evening. I know while testing it I had some posts go through, while others would give me a DB error.
I think we are in the clear now. To be safe, in addition to the full backups that take place overnight, I am adding one more backup of the forum database during the day. So in addition to the forum backup at 10am and 10pm central time, I have now added a backup at 4:00 Central time. While these database backups are going on, you could have the forum "hang" for 30-60 seconds or so, where your page won't load, because it needs to get a clean backup, so it pauses database requests.
Sorry for any headaches this caused last night.