Thoughts on Hardware Fail-Over / Load Balancing

[JGadrow]

Ok, my boss has asked me to put together a white paper for some possible solutions. Basically, he's wanting to rent 2 servers (hosting is already determined) and provide a fail-over solution. He would like to include load balancing if possible.

However, I'm not seeing much potential benefit for load balancing because of the multi-master situation. I'm thinking the best solution, at this time, is to simply employ a backup server to provide for fail-over. Does everyone agree that this is the 'best' possible solution to implement with a 2-server setup?

Thanks in advance for your input!

[Harkins]

What work are the servers performing? You say "multi-master", so sounds like MySQL?

[Chris]

Load balancing for 2 servers? I don't see the benefit. If there were like 5 servers... With 2 you usually split them into DB/everything else.
You could also suggest raid harddisk to increase reliability.

[Delifisek]

It depends.

What is your exact need ?

[JGadrow]

Ok, a bit more details: Both machines would have contain webserver and mysql server because failover is the critical portion so having separate data / app servers isn't viable at this time. Currently, they have 4 servers which all 'stand-alone' with various client websites running on them. They're attempting to consolidate them into a single server so that they can decommission 3 of the 4 because they operate on outdated technology (these are rented servers, not owned). Then, they're wanting to commission a 2nd server to provide redundancy for fail-over. My boss wanted me to write a white paper discussing the possibility of load balancing the applications to also increase performance while we're gaining this redundancy.

However, the data is too large to store in RAM so MySQL Clustering is out leaving only Replication as viable. Since we only have 2 servers, we cannot store data in a master server and slave it back out to the app servers and having both servers write to a single server and slave to the other would, likely, provide no real load balancing. I am not 100% certain of that, but I imagine that the server performing all of the write operations would slow to a crawl due to constant writes.

Basically, they're not 100% against the idea of having multiple servers, but the goal is to reduce operating costs if possible. Load balancing isn't really an issue at this time but would be a 'nice feature' if it could be provided. That's why, I believe, having a single 'backup' server behind the load balancer ready to come online if the primary server goes offline is the best scenario.

[Scion]

At my previous employers we developed and administered a web bassed application. We did have a seperate DB and Web Servers but that was to achieve better performance. However if i simplify things and bit we could just say that the whole stack was mirrored. Our hosting provider had a solution that worked a little like a heart beat test in that as soon as it identified that a single machine was unavailable it switched over to using the mirror. During the switch over the web app would be unavailable (with splash screen to say back in 5) I can t remember exactly how long the switch over took, certainly most times we did it manually no users ever noticed anything.

In the 4 years i was there i think we only had 1 or 2 incidences when the mirror was put into use automatically, but as ive already indicated it did have a lot of value in other ways, for example OS upgrades or software updates all could be done on one machine and the mirror be available for rollback if necessary.

If your looking to try and make as much use out of your resource as possible then having a mirror that just sits there waiting doing nothing much untill its needed is a rather expensive option. It doubles your hardware costs for no extra performance (in fact the process of maintaining the mirror actually degrades performance somewhat)

However i can see a couple of other options that could be considered.

firsty split the DB and the web server.....have one on each machine with its mirror on the other....that way your maximising your use of the two machines and should one fail then you switch the master that was on that machine to the mirror on the other and your back to operating the stack on one single machine.

another option is to put the master DB on one machine and mirror it on the second and then set up a webserver cluster across the two available machines. if the machine with the master DB fails then you switch to the mirror and your web server cluster contracts to a single node.

Im sure there are other options as well...just remember to put in your rational an estimate of the incurred costs of downtime, depending on the situation even being able to stop 24Hrs of downtime may well be worth the cost of the additioanl hardware for 3 years.

[JGadrow]

Thanks for the detailed write-up. However, I was given the wrong impression about what was required for 'fail-over' on this. Apparently, they want 'automatic fail-over.' Meaning if Data Center A is wiped out by an electrical fire (as has occurred in the past), then Data Center B is still online and operational.

This is, basically, a service for the clients. The company I work for provides hosting to some clients (they generally don't like to but some clients just find it easier) and they want to offer for automatic failover to meet 99.999 uptime requirements.

Server load isn't really an issue, it was just an option they wanted me to explore. I am going to monitor how the server actually performs under a load and the addition of replication. If we experience more processing than a single server can handle, they're prepared to go back to our original number of machines (4) but, obviously, any costs that are unnecessary should be avoided.

Plus, I'd rather have them spending the extra server budget on my development environment! lol

[Scion]

It probably also depends heavily on the hosting arrangement you have....our servers were self managed...to an extent....backups and any physical upgrades were handled by our hosts, but we managed the actual servers....(i use we in the sense of our Sys Admin ...not me ) This made sense because we were doing weekly if not more frequently deployments that we would have otherwise had to put through the hosting company.

Our hosting company had two physical locations, I know we had the option to go for geographical-redundancy for the machines.... but just chose to do it with the backups...

you should be able to get a hosting packages that offer this to you seamlessly.....sounds like you guys should sit down with your account manager from the hosting provider and spell out what you want....if you dont have an account manager....then switch to a hosting provider that will provide one.

[JGadrow]

We have an existing relationship with a hosting provider that my boss would like to continue because they have been excellent with the customer service. It's a hosting provider called, "The Planet." Never heard of them before I came here.

I'm not 100% positive that they're the "best" solution for us but I'm kind of stuck with them for now. From what I've gathered (and I might be wrong in this) but they handle the server and software on it (which bums me out because I hate having to deal with the hosting provider to make changes to server config but *shrug* what's one to do? lol). However, they don't provide guarantees on operation. Really, in my opinion, it should be The Planet's responsibility to make sure that the servers that we're paying for are up and running at all times, but I suppose that's not part of the deal. So, to guarantee that ability for our clients we have to secure the additional hardware for replication / etc. ourselves.

But, I digress. I have lots of 'pie in the sky' wishes that can't come true right now. For the time being, I have 2 servers and I have to guarantee for automated fail-over. I really don't see any other option than Server A fully mirrored by Server B behind a load balancer that adds Server B to the pool if Server A has no heartbeat and co-locating the servers in different geographic locales.

I need to check, however, what happens if the power goes out in Data Center A and Data Center B comes online and then Data Center A comes back online. Does Data Center B get removed from the server pool and replaced automatically by A? Do I have to manually reset the pool? Etc. Obviously, I want to avoid a situation in where Server B comes online, handles 100 write operations and then Server A is re-added to the pool by the load balancer before I've had a chance to synchronize the data between them.

[Delifisek]

And where is your load balancer ?

Who routes your packets from DC A to DC B ?

[JGadrow]

That's an additional piece of hardware run by The Planet. I believe when I asked about it I was told that the load balancer did have a redundant backup in another data center in case it went offline. Because you wouldn't want to simply make the load balancer the single point of failure either.

Basically, we list the DNS record with an IP and that address is handled by one of the load balancers that'd be allocated to us. It is forwarded to the machine in Data Center A unless that center is offline in which case it goes to B. As far as I know, on our end, there is no single point of failure with this setup. Which is the primary goal.

Of course, I intend to work on them to create a separate data server but I don't think that's a battle I'm likely to win at this point even though it is agreed to be a 'best practice' for us.

[Delifisek]

Scion's offer seems fit in this scenario.

Build MySQL master slave config. Do live update on your backup (via mysql master slave). And load balancer will do the rest.

[JGadrow]

The only issue with that is that the app servers would be written to write to the master database server which creates a single point of failure again. However, I suppose I could contact the host and see if there's a way to do the database connections through the load balancer as well. In which case, the app server's connection to the database is already abstracted behind a load balancer making it totally unaware of where data is going to...

Hmmm... that's worth considering.