Author Topic: Request for AH to be up and available! (Read 1207 times)

grmrpr · « **on:** June 13, 2005, 01:53:31 PM »

AH-

After another weekend of outages due to Savis is there any hope of having a reasonable expectation that AH will be up and running at any given point? I do not know what the SLA is with Savis but I do not think they are capable of delivering. Isn't it reasonable to expect at least five 9's (99.999%) uptime out of this service we all pay for?

I personally do not have experience with Savis but I can tell you I have never heard anything positive about them. I have used hosting facilities such as: UUNET, Digex, GenuityNet, AT&T, Inflow (Now SunGuard) and I have never seen so many outages as I have seen with AH since it has moved to Savis.

What’s the plan here? Or are we (The players) just expected to live with the shotty service being provided by Savis?

As a network engineer and unix sysadmin here are some other thoughts/suggestions.

Is it feesable to run backup bandwidth into your systems and use BGP to route the traffic when there is a failure. I have heard statements that the amount of data the end users need is relatively small. Could a single or maybe even multiple T1's be purchased through another bandwidth provider and ran into your cab's at Savis? This way when Savis goes down you will have another route into the servers.

Also may I suggest fixing your DNS. hitechcreations.com lists 2 ip's as DNS providers. Both the IP's are on the same network. When your network goes down so does your DNS. I would suggest listing 2 or three more. What I typically do is have my service provider slave my tables from me but actually list my providers dns servers as primarys with internic. That way their servers take all the hits and I can update my tables at will on my own server without going through them. Of course list your servers as additional DNS servers...

Have you considered moving the website off that network and away from the game machines? I personally would consider moving it to another data center (A data center close to the main office could be an option) and using someone like Mirror Image or Akamai to deliver the downloads for you so that bandwidth is kept away from the game servers. I am sure the stats stuff play a role in the decision process but all that backend stuff could still be they way it is. If for whatever reason that portion goes down at least HTC web presance would still exist and be able to provide notifications of any outages or issues....

Oh well just some thoughts and rants about my dissapointment with the avialability of AH. I hope no offense is taken and if there is anything I could do to help out with a solution please let me know.

GrmRpr

Karash · « **Reply #1 on:** June 13, 2005, 06:06:05 PM »

Constructive Post!

Thumbs up!

MOIL · « **Reply #2 on:** June 13, 2005, 11:23:57 PM »

OH!!! this is real smart, start talking to make sense and offer up solutions or ideas that might help!!!!

Are you crazy:confused: what if other people start helping or offering such constructive insight into a problem...................... ....................IT WILL BE PURE PANDEMONIUM.................. .......It's insane I tell ya........Insane:eek:

SuperDud · « **Reply #3 on:** June 14, 2005, 12:19:45 AM »

You mean instead of throwing a fit and saying I'll take my $15 elsewhere? You can actually try to be helpful? Who knew???

Wolfala · « **Reply #4 on:** June 14, 2005, 12:49:51 AM »

What i'm curious is this: how much data is being pushed during a 400-500 pilot login? For our ATC servers over at Westcoastatc.com, we purchase the OC3 line directly from AT&T while bypassing the Savis and Verio's completely, b/c AT&T owns the lines that the others rent.

Skuzzy,

What are a few solutions?

Wolf

SFCHONDO · « **Reply #5 on:** June 14, 2005, 01:36:29 AM »

Nice Post, Would love to see Skuzzy reply to this one.

onions4u · « **Reply #6 on:** June 14, 2005, 03:03:36 AM »

Skuzzy, I live in Texas 31 to 32 ping rate lately, Squaddie lives in Washington state and lately he's having 200 ping rate and getting discoed alot. Both have cable, I have comcast he has one called (3d?) another company.

grmrpr · « **Reply #7 on:** June 14, 2005, 08:25:50 AM »

At my last job I managed a network that consisted of 7 warehouses across the country. Each facility had private frame relay and T1's that were provided by AT&T. Also each facility had third connections as in cable or DSL. We load balanced all those lines across a device called FatPipe AT&T owns a really solid network and with our redundancy we had virtually 0% down time. We ran VOIP which I would say is even more sensitive than the UDP game traffic. There are relatively inexpensive ways to solve the problems HTC are having.

I wish HTC would respond with a clue as to what their network architecture is and what usage requirements are so the engineers in the community could offer free advice. Also I have negotiated bandwidth deals with all the major carriers and have toured all the facilities I have listed. If HTC would make available the architecture and some small details I am sure the techies in the community could offer viable solutions to improve the service.

But alas HTC seems to be silent on this issue…

GrmRpr

hitech · « **Reply #8 on:** June 14, 2005, 09:13:08 AM »

grmrpr: All items you have listed have been considered previously and are beeing considered again.

HiTech

grmrpr · « **Reply #9 on:** June 14, 2005, 09:45:07 AM »

HiTech-

If there is any assistance I can provide please do not hesitate to contact me. I would be more than willing to review any RFP's or support Skuzzy or HTC in any capacity. I will send you and Skuzzy a copy of my resume so you can see if I have any skill sets I could volunteer that may be helpfull to the cause.

Regards-

GrmRpr

hitech · « **Reply #10 on:** June 14, 2005, 10:44:05 AM »

Looks like we are switching to at&t data center in dallas. It has one nice advantage in that it has dual oc48's from different areas.

Only issure on configuration I have not been able to solve is how to do source routing with solaris. It just saves me having to put in a router at the data center. I would then be able to do user selectible routes to the data center.

HiTech

grmrpr · « **Reply #11 on:** June 14, 2005, 11:15:50 AM »

Am I assuming mutliple interfaces in the Solaris box? Can you explain source routing a little better? I am assuming you are meaning if source =x go out interface y. I had a similar delima that I ended up addressing with an Extreme Switch. I had hosts coming in from differant networks and wanted to split the traffic on the multiple interfaces.

Basically this is what I did.

In the extreme switch (Alpine 3804) I created a psuedo load balanced IP and had the down stream clients talk to it. That load balanced IP used latency alogorithims to determine which interface on the E450 Solaris 9 machine was the least used and communicated down that. It also provided redundancy in that if a interface went down on the sun box traffic still flowed.

Only issues. In a failure of an interface the traffic associated to that interface died due to limitations in the stack on the sun box. It could not switch the traffic between interfaces. But all traffic was able to reconnect with out a problem. Unless there is some sort of logic on the server side I do not know how the server would deal with traffic switcing interfaces on the machine. The switch would keep the traffic persistant as long as an interface didnt fail but if it did you may wind up loosing the traffic that was associated with the failed interface.

GrmRpr

Side note-

The Extreme gear has all the old big IP stuff built into it. I like Cisco for most needs but when it comes to HA servers I've found the cheapest method is to put a Extreme Layer 3 Switch in place. They are so much easier to configure and manage. And frankly Cisco has never been strong in load balancing.

grmrpr · « **Reply #12 on:** June 14, 2005, 11:23:21 AM »

PS what is the firewall solution?

If you had a PIX 520 in place in front of this you could add tools to the server side to send shunts to the PIX to block any ip deemed malicious.

Just for example say user X gets really out of hand and starts going nuts. Maybe this user has some basic script kiddy attack skills. An admin could type a server command that would send his IP to the PIX to block it.

Karash · « **Reply #13 on:** June 14, 2005, 11:58:14 AM »

Good to see this is getting worked on. Thanks HT!

grmrpr · « **Reply #14 on:** June 14, 2005, 12:37:11 PM »

Dale-

Something like this:

http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&category=51268&item=5782073294&rd=1

But with the port density you are talking about prob about 1/2 that cost. And you can get Extreme to recertify and brought into maintenance if you buy a support contract with them.

GrmRpr