Moving BroncosForums to new server - view thread for details of timing and downtime

[Tned]

Ok, I'm in the process of finalizing an order for a substantially beefed up server. The load on the server had been slowly increasing since last summer, but there was still quite a bit of headroom. However, vBulletin 4.x puts a much greater load on the server, and as a result we are getting periodic slow downs. I knew this was a possibility before upgrading, but as I've stated elsewhere, it was time to upgrade vBulletin for a large number of reasons.

The new server will have a substantially faster processor and substantially faster hard drives. I'm going to keep the same amount of RAM initially, although it is faster RAM, but if needed it will support more RAM and they can upgrade that in the middle of the night with only a few minutes of downtime.

If I get all the details finalized tomorrow, then I will try and do the migration Friday night/Saturday morning or possibly Saturday night/Sunday morning. If I miss that Window, then the next one will probably be next Saturday night/Sunday morning.

While I will do as much of the migration as possible in the overnight hours, there will be possibly an hour or so of actual downtime. Beyond that, the server IP will change so it will require a DNS change. How quickly this occurs will be based on whether or not your ISP follows the TTL setting of the BroncosForums DNS server, or ignores it as some ISPs do and simply updates their DNS every 4, 12 or 24 hours.

Once the old forum is shutdown and the new one is live, I will have a message on the old one with a direct IP address link that can be used in the short term until the DNS is fully propagated.

Anyway, it will be a relatively short disruption, but should put us in much better position to handle any surge around the draft or once things pick back up when the season starts and the Broncos make their SB run.

If you have any questions you can post them here.

T

[underrated29]

What the hell did you just say?

[wayninja]

I'm waiting for someone to blame this on Tebow.

[DisturbedShifty]

I'm waiting for someone to blame this on Tebow.

I blame Tebow!

Also, does that mean the new forum will have a different look to it?

[sneakers]

We are ******!

[sneakers]

Ok, I'm in the process of finalizing an order for a substantially beefed up server. The load on the server had been slowly increasing since last summer, but there was still quite a bit of headroom. However, vBulletin 4.x puts a much greater load on the server, and as a result we are getting periodic slow downs. I knew this was a possibility before upgrading, but as I've stated elsewhere, it was time to upgrade vBulletin for a large number of reasons.

The new server will have a substantially faster processor and substantially faster hard drives. I'm going to keep the same amount of RAM initially, although it is faster RAM, but if needed it will support more RAM and they can upgrade that in the middle of the night with only a few minutes of downtime.

If I get all the details finalized tomorrow, then I will try and do the migration Friday night/Saturday morning or possibly Saturday night/Sunday morning. If I miss that Window, then the next one will probably be next Saturday night/Sunday morning.

While I will do as much of the migration as possible in the overnight hours, there will be possibly an hour or so of actual downtime. Beyond that, the server IP will change so it will require a DNS change. How quickly this occurs will be based on whether or not your ISP follows the TTL setting of the BroncosForums DNS server, or ignores it as some ISPs do and simply updates their DNS every 4, 12 or 24 hours.

Once the old forum is shutdown and the new one is live, I will have a message on the old one with a direct IP address link that can be used in the short term until the DNS is fully propagated.

Anyway, it will be a relatively short disruption, but should put us in much better position to handle any surge around the draft or once things pick back up when the season starts and the Broncos make their SB run.

If you have any questions you can post them here.

T

Tned:


http://bagelfish.files.wordpress.com/2011/04/wow-nerd-south-park-580.jpg

[Northman]

Tned:


http://bagelfish.files.wordpress.com/2011/04/wow-nerd-south-park-580.jpg


Get it right Sneak.

http://1.bp.blogspot.com/_dJylj_lumkc/Sw6be8EdSQI/AAAAAAAAAEk/zO1YVpaoKzQ/s1600/really-fat-guy-on-computer.jpg

[Tned]

Cat boy's is more accurate, but without the wrist brace. I think he's been stalking me...

[Tned]

I just placed the order. If it's online and ready to go tonight, I'm going to try and migrate it over night. If not, I'm shooting for Sunday night. The way they are recommending I migrate, I won't have the older server online like last time, so the server could be offline for 2-6 hours, depending on how quickly your ISP updates their DNS info.

If we go with this method, I will email everyone the IP address that can be used on a temp basis.

I'll post another update this afternoon, once I have the timeline nailed down.

[Dapper Dan]

2-6 hours? What the hell will I do? Ok. I'll make another mock draft. Maybe a full 7 round mock for each team.

[Tned]

Ok, they don't have the drives I want in stock, but have some on order that will be at the data center on Monday. So, I will probably not do the server move until Saturday night/Sunday morning, as I work all week and will be tied up all day Saturday at fantasy baseball drafts.

There is a small chance I will decide to go ahead with the migration during the week, if so, it would probably be Tuesday or Wednesday around Midnight.

Updates to follow.

[Joel]

I'm waiting for someone to blame this on Tebow.
EFX haven't shown the ability or desire to make this site truly competitive. They should crack open their wallets for real talent like The Nostalgia Critic or Anonymous, because scrubs like Zam and Cugel will never make us contenders. :tongue:

[LTC Pain]

Thanks Tned :)

[Tned]

Just an update.

The new server was built today, but some things weren't exactly the way I had spec'd it (the actual RAID card, disk partiioning, etc.), so they are rebuilding it. I'll spend the rest of the week configuring it, and then the plan will be to move BroncosForums to the new server Saturday night/Sunday morning, probably around 1:00am when I get home from my baseball draft.

For those of you on now, the new server should hopefully eliminate the sluggish response in the early AM hours when several different backups are being done, as well as speed things up across the board (at least that's the hope).

[Canmore]

I wondered what was going on with the upgrade. Thanks and good luck.

[Tned]

I wondered what was going on with the upgrade. Thanks and good luck.

They didn't have the hard drive I wanted (Intel 520 SSD - 180GB) in stock last week, and they didn't get some more in until today. Then, it was loaded wrong. Should be ready to start working on the config and stuff by tomorrow.

[Canmore]

They didn't have the hard drive I wanted (Intel 520 SSD - 180GB) in stock last week, and they didn't get some more in until today. Then, it was loaded wrong. Should be ready to start working on the config and stuff by tomorrow.

How did they load the drive wrong?

[Tned]

How did they load the drive wrong?

Partitioning. The current server 4 drives in a Raid 10. The new drive is going to have two SSD drives in a Raid 1 (faster than sata in a raid 10) and two 500 GB normal Sata drives in a Raid 1.

My plan was to have the second raid just used for backups, or possibly some unpartitioned space if I ran out of space on the SSDs.

Then, I wanted the Linus OS, /var directory (where MySQL is located), /home directory (where web files are located), etc., all on the MUCH faster SSD raid.

What they did was put most of the OS stuff on the raid, which was good, but put the /home directory (contains the web pages, php code, etc.) on the slower SATA raid.

In reality, it might have been fine. Most of the PHP files and other web files would probably stay in cache most of the time, but I increased the up front and monthly of the server quite a bit to move up to the new Intel 180GB 520 SSD drives, because they are considered about the fastest around right now (the smaller ones are slower), so I wanted everything but the backup partitions on the SSDs to try and eliminate an I/O bottlenecks.

That might have been a longer answer than you were looking for, or said in a much shorter way "they partitioned the drives wrong" ;)

[Canmore]

Partitioning. The current server 4 drives in a Raid 10. The new drive is going to have two SSD drives in a Raid 1 (faster than sata in a raid 10) and two 500 GB normal Sata drives in a Raid 1.

My plan was to have the second raid just used for backups, or possibly some unpartitioned space if I ran out of space on the SSDs.

Then, I wanted the Linus OS, /var directory (where MySQL is located), /home directory (where web files are located), etc., all on the MUCH faster SSD raid.

What they did was put most of the OS stuff on the raid, which was good, but put the /home directory (contains the web pages, php code, etc.) on the slower SATA raid.

In reality, it might have been fine. Most of the PHP files and other web files would probably stay in cache most of the time, but I increased the up front and monthly of the server quite a bit to move up to the new Intel 180GB 520 SSD drives, because they are considered about the fastest around right now (the smaller ones are slower), so I wanted everything but the backup partitions on the SSDs to try and eliminate an I/O bottlenecks.

That might have been a longer answer than you were looking for, or said in a much shorter way "they partitioned the drives wrong" ;)

No, actually I was curious. I've been a system administrator both in and out of the Marine Corps. I have a minor/near major in Computer Science along with a bunch of other stuff. I used to build my own computers also. It's interesting to me.

[Tned]

No, actually I was curious. I've been a system administrator both in and out of the Marine Corps. I have a minor/near major in Computer Science along with a bunch of other stuff. I used to build my own computers also. It's interesting to me.

Ahhh. Gotcha.

I've been hesitant to take the plunge on SSD's, but since the only real bottleneck for BF is being I/O bound at times, and I've read articles showing that a single very fast SSD is as fast as 8 to 13 (depending on whether it's random read/write, sequential, etc.) SCSI/SAS 15k drives in a RAID 0 setup. These things are fast.

[GEM]

I hope this helps with the speed on my phone. It's not an Iphone or a Droid, just a samsung with browsing abilities. I didn't have any trouble before the Vbull update, was fast enough. Now it's so slow, I don't even bother on the weekends. Hope this helps!!

[Slick]

I hope this helps with the speed on my phone. It's not an Iphone or a Droid, just a samsung with browsing abilities. I didn't have any trouble before the Vbull update, was fast enough. Now it's so slow, I don't even bother on the weekends. Hope this helps!!

Same for me on the Blackberry. It works, but it's so slow it's painful, but I think it's on my end. If I can ever get time to go up to the store, I am eligible to upgrade. My BB uses Edge network, which I think is one of the slowest. We still don't have 4g here but there is 3g. Hopefully that will improve the mobile site for me.



Thanks for the hard work Tned!

[Tned]

I hope this helps with the speed on my phone. It's not an Iphone or a Droid, just a samsung with browsing abilities. I didn't have any trouble before the Vbull update, was fast enough. Now it's so slow, I don't even bother on the weekends. Hope this helps!!

I'm hoping so as well. Are you using the full site or the new mobile site/style from your phone?

The server is definately getting taxed at times and when it does, it's slow.

I'm running a benchmark on the new server right now to get a feel for how much faster the hardware is than the old one.

[GEM]

I'm hoping so as well. Are you using the full site or the new mobile site/style from your phone?

The server is definately getting taxed at times and when it does, it's slow.

I'm running a benchmark on the new server right now to get a feel for how much faster the hardware is than the old one.

I wasn't aware the mobile was back. I thought that was gone with the upgrade. I know there is a Droid and an Iphone, but I don't have either of those phones. If there is a new mobile, I will give that a shot.

[WARHORSE]

Whats a server?


I'll take a Kamakaze with a prune in it.

[Tned]

I wasn't aware the mobile was back. I thought that was gone with the upgrade. I know there is a Droid and an Iphone, but I don't have either of those phones. If there is a new mobile, I will give that a shot.


Yep, it's back and better. You can go down to the bottom left and choose it from the list of styles (same way you switch to worker friendly) or http://mobile.broncosforums.com/forums/forum.php is working again.

[Tned]

I'm doing an initial migration of data from the old server to the new one, in order to make sure everything is configured correctly. That will cause the site to run sluggish at times.