UPDATE 9/9/13: See original post below.
Ok, I know a little more about what happened. A relatively recent security problem with vBulletin was discovered, where if the install directory was not deleted after an install/upgrade, it allowed a hacker to create an administrator account. For years, vBulletin didn't recommend deleting this directory, but instead only the install.php file. That changed recently. A few weeks ago they posted a notice about a possible exploit on their support site telling people to delete their installs directory, but didn't send an email. Then, apparently last week, a TON of sites got hacked all at once, so they sent an email on 9/3 with the same information that had been on their support site.
I was tied up with work, and still recovering from my house flood, so not keeping up with email. So, I didn't see it until days later. As it turns out, they had created an admin account on the 3rd on BF, so even if I had read the email and deleted the install directory right away, it would have been too late as they already had access, because in one fell swoop they compromised hundreds or thousands of vBulletin servers, then days later, started messing with them.
I have been working with the server company, my old support company that I'm not under contract with anymore but paying hourly, and vB support, and we believe we have covered all the bases, but vBulletin is being pretty tight lipped about all the details, as they try and determine the best way to avoid it in the future (besides just deleting the install directory).
I'm also considering changing server companies, even though I've been with this one for five years. While the hack had nothing to do with them, their response time and some other issues were unacceptable. If I decide to do it, I will probably do it next Monday, and the actual downtime should be minimal, these days DNS usually updates quickly, and it should only be 30 minutes to a few hours. My work schedule is nasty now, and then I have one week where it's not to bad, then I'm traveling part of the week of the 23rd, week of the 6th, and then out of the country for 15 days starting 10/15, so I need to have datacenter that will react quickly if we have another problem.
I'll post another update in a day or two if I learn more about what happened, or decide to move ahead with the datacenter change.
=========================
Ok, what happened?
The server was hacked yesterday. In order to be 100% careful, I immediately took BroncosForums down, while the support people investigated what happened. Once it was determined that the server was definitely compromised, then the decision was made to rebuild the server from scratch.
First, we had to have a gameplan for rolling all files back to a known good point, but not losing and data/posts. Then, it was a very slow process to clone the old drives for a backup, reinstall, reload, etc. I grabbed a couple hours sleep and got up at 3:00 am, and worked with the support people to finish, but it still took them until nearly 9:00 before they had finished their part of the server reload, security hardening, etc.
I then started the backup restore, and then we had some major problems getting that to work on the newly installed server (you probably saw the pages very messed up when you first saw the server come back up). That took many hours to figure out.
Anyway, we are now on a freshly reloaded server, with the latest vBulletin and other software.
I’ll provide further details and answer any questions later, but now I have to travel about 1 hour to a work training that I’m very late for. It will go until midnight or so, but I’ll be keeping my eye on things via my phone.