been warping and disco'ing about 4 wks now

[yanksfan]

I have ck'ed everything i know to ck, my system is clean as far as i can tell, rolex suggested i come here after running a traceroute, i did that and comcast has a hub in NY (go figure) that looses packets at a rate between 14% to 60% .
{ 4.71.186.14 } <----This is the culprit, could this be my problem, I have run the trace several times and this is the only one that looses packets.

Any light you could put on this for me would help as the game is getting unplayable for me, lag, rubber bullets, warp, and disco are just making me crazy, not that i have far to go, but ya know.

Thanks   Yanks,   (Don)

[REP0MAN]

Can you post the ping plotter results for us to look at?

[yanksfan]

Originally posted by REP0MAN
Can you post the ping plotter results for us to look at?

Here they are, one right after the other, i'm actually pretty happy i figured out how to do that

[REP0MAN]

I'm sure Skuzzy will agree, your problem is Level3. Everything else looks decent. Level3 has issues and I'm not sure there is anything you can do about it. I'd show these results to Comcast first and see where that gets you.

[yanksfan]

Originally posted by REP0MAN
I'm sure Skuzzy will agree, your problem is Level3. Everything else looks decent. Level3 has issues and I'm not sure there is anything you can do about it. I'd show these results to Comcast first and see where that gets you.

Thats pretty much what i was thinking, thanks for looking at them, i will collect a few more and call comcast, but you know they aren't interested in fixing it.

Don

[Skuzzy]

It is intersting to note it happens to be the gateway between Comcast and Level3 where the problems are.  Definately get this to Comcast, and good luck.

[REP0MAN]

I agree Skuzzy. Tracing from my home network to yank's IP, I get smooth sailing up to his local Proxy. Of course their is an absence of one particular company....

traceroute to 73.180.246.1 (73.180.246.1), 64 hops max, 40 byte packets
 1  192.168.1.1 (192.168.1.1)  1.206 ms  0.835 ms  0.763 ms
 2  ip70-176-8-1.ph.ph.cox.net (70.176.8.1)  13.780 ms  11.177 ms  11.632 ms
 3  ip68-2-6-69.ph.ph.cox.net (68.2.6.69)  9.805 ms  17.931 ms  8.862 ms
 4  68.2.13.94 (68.2.13.94)  9.282 ms  8.301 ms  9.397 ms
 5  68.2.13.9 (68.2.13.9)  9.997 ms  13.589 ms  30.431 ms
 6  68.2.13.5 (68.2.13.5)  11.381 ms  15.025 ms  13.527 ms
 7  68.2.13.1 (68.2.13.1)  14.330 ms  11.942 ms  12.541 ms
 8  chnddsrj01-ae2.0.rd.ph.cox.net (68.2.14.9)  22.047 ms  7.418 ms  9.635 ms
 9  langbbrj02-as0.r2.la.cox.net (68.1.1.231)  27.534 ms  24.455 ms  28.735 ms
10  te-0-7-0-4-cr01.losangeles.ca.ibone.comcast.net (68.86.84.53)  22.899 ms  25.202 ms  24.702 ms
11  te-0-7-0-0-cr01.stratford.tx.ibone.comcast.net (68.86.84.57)  51.370 ms  53.840 ms  49.492 ms
12  te-0-7-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.84.61)  61.507 ms  73.486 ms  61.973 ms
13  68.86.85.65 (68.86.85.65)  81.235 ms  82.514 ms  79.100 ms
14  68.86.85.73 (68.86.85.73)  88.415 ms  82.686 ms  101.531 ms
15  68.86.85.5 (68.86.85.5)  79.414 ms  81.671 ms  80.071 ms
16  te-0-4-0-6-cr01.philadelphia.pa.ibone.comcast.net (68.86.90.9)  83.524 ms  86.026 ms  84.749 ms
17  po-90-ar01.401nbroadst.pa.panjde.comcast.net (68.86.208.29)  88.163 ms  89.703 ms  88.636 ms
18  po-80-ar01.wallingford.pa.panjde.comcast.net (68.86.208.30)  92.787 ms  89.170 ms  87.168 ms
19  te-2-3-ar01.audubon.nj.panjde.comcast.net (68.86.210.97)  90.181 ms  89.658 ms  91.431 ms
20  te-1-1-ur01.woodbury.nj.panjde.comcast.net (68.86.210.98)  91.304 ms  89.799 ms  89.658 ms
21  te-1-1-ur02.woodbury.nj.panjde.comcast.net (68.86.210.102)  92.934 ms  90.616 ms  93.465 ms
22  te-1-1-ur01.swedesboro.nj.panjde.comcast.net (68.86.210.106)  92.472 ms  89.453 ms  89.490 ms
23  te-1-1-ur01.franklinvill.nj.panjde.comcast.net (68.86.210.110)  90.860 ms  88.307 ms  89.650 ms


Good Luck Yanks!

[yanksfan]

Originally posted by Skuzzy
It is intersting to note it happens to be the gateway between Comcast and Level3 where the problems are.  Definately get this to Comcast, and good luck.

Thanks guys, i will talk with comcast, but for all the trouble i have had with them i'm sure i will be looking into a dsl, comcast is not interested in updateing their system, only loading it up with new acct's.

i don't know how they are in the rest of the U.S. but they are really bad here in jersey, i really appreciate you guys lookin at this for me, Thanks.

Don

[DerHelm]

Originally posted by REP0MAN
I agree Skuzzy. Tracing from my home network to yank's IP, I get smooth sailing up to his local Proxy. Of course their is an absence of one particular company....

traceroute to 73.180.246.1 (73.180.246.1), 64 hops max, 40 byte packets
 1  192.168.1.1 (192.168.1.1)  1.206 ms  0.835 ms  0.763 ms
 2  ip70-176-8-1.ph.ph.cox.net (70.176.8.1)  13.780 ms  11.177 ms  11.632 ms
 3  ip68-2-6-69.ph.ph.cox.net (68.2.6.69)  9.805 ms  17.931 ms  8.862 ms
 4  68.2.13.94 (68.2.13.94)  9.282 ms  8.301 ms  9.397 ms
 5  68.2.13.9 (68.2.13.9)  9.997 ms  13.589 ms  30.431 ms
 6  68.2.13.5 (68.2.13.5)  11.381 ms  15.025 ms  13.527 ms
 7  68.2.13.1 (68.2.13.1)  14.330 ms  11.942 ms  12.541 ms
 8  chnddsrj01-ae2.0.rd.ph.cox.net (68.2.14.9)  22.047 ms  7.418 ms  9.635 ms
 9  langbbrj02-as0.r2.la.cox.net (68.1.1.231)  27.534 ms  24.455 ms  28.735 ms
10  te-0-7-0-4-cr01.losangeles.ca.ibone.comcast.net (68.86.84.53)  22.899 ms  25.202 ms  24.702 ms
11  te-0-7-0-0-cr01.stratford.tx.ibone.comcast.net (68.86.84.57)  51.370 ms  53.840 ms  49.492 ms
12  te-0-7-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.84.61)  61.507 ms  73.486 ms  61.973 ms
13  68.86.85.65 (68.86.85.65)  81.235 ms  82.514 ms  79.100 ms
14  68.86.85.73 (68.86.85.73)  88.415 ms  82.686 ms  101.531 ms
15  68.86.85.5 (68.86.85.5)  79.414 ms  81.671 ms  80.071 ms
16  te-0-4-0-6-cr01.philadelphia.pa.ibone.comcast.net (68.86.90.9)  83.524 ms  86.026 ms  84.749 ms
17  po-90-ar01.401nbroadst.pa.panjde.comcast.net (68.86.208.29)  88.163 ms  89.703 ms  88.636 ms
18  po-80-ar01.wallingford.pa.panjde.comcast.net (68.86.208.30)  92.787 ms  89.170 ms  87.168 ms
19  te-2-3-ar01.audubon.nj.panjde.comcast.net (68.86.210.97)  90.181 ms  89.658 ms  91.431 ms
20  te-1-1-ur01.woodbury.nj.panjde.comcast.net (68.86.210.98)  91.304 ms  89.799 ms  89.658 ms
21  te-1-1-ur02.woodbury.nj.panjde.comcast.net (68.86.210.102)  92.934 ms  90.616 ms  93.465 ms
22  te-1-1-ur01.swedesboro.nj.panjde.comcast.net (68.86.210.106)  92.472 ms  89.453 ms  89.490 ms
23  te-1-1-ur01.franklinvill.nj.panjde.comcast.net (68.86.210.110)  90.860 ms  88.307 ms  89.650 ms


Good Luck Yanks!

Yeah, but that company (Level 3) should be there.
That route Cox is using is going from Phoenix to LA to Tx .. to NJ.
Cox has Level 3 in Phnx, MCI, WV Fiber and a few other transits in that location.
The Level 3 path is 20 ms shorter than that one currently in your routing tables.


Now about your ping plot though.

Keeping in mind it was still only ran for 2 minutes.

Couple thing you should be corrected about that.


1) That is Level 3's problem

Skuzzy is correct that this is the meet point between Level 3 and Comcast.
To be more specific, that router with the IP 4.71.186.14 is Comcasts. The IP is owned by Level 3 but it is provisioned on Comcasts router. When just 2 routers are directly connected, in this case for Paid Transit, only 2 usable IPs are needed so a /30 block is cut out.

4.71.186.12 = Network ID (non usable)
4.71.186.13 = Usable Level 3 side
4.71.186.14 = Usable Comcast Side
4.71.186.15 = Brodcast IP

Now the 2 interconnected routers can communicate unicast. If this was a Peer pool or the like, a bigger block would be needed.


2) That hop 9 is the problem (IP 4.71.186.14)

It's actually not.  I'll explain where the problem is down the line, but first I think I have to correct why this is not the problem.

First thing about packet loss you have to alway remember is:
If it's true packet loss, every hop passed that router would be affected including the destination network. (special cases withstanding, but this is not one of them)

So if there was 10 % packet loss on hop 9, then 10 -22 would look just as bad.  But as you can see it is only occuring in 2 places, hop 9 and 21. From with in AT&T's network only, I can reproduce that 10% packet loss that is showing up on hop, but it will not show on hop 22. That tells me that hop 21 is not an actual problem. A packet is not like cars that speed back up.
If it is slowed down and 10 seconds is added to the journey, then that is an extra 10 seconds from that point on till it reaches it destenation. It may have better luck on that next packet though. And when you lose a packet on router's interface card, it doesn't matter where it is going to.


OK, now with that in mind, I do see a problem. But from what is shown here, it's in hop 11, 12 or 13. Again keep in mind this was only ran for 2 minutes and sampled 10 times.  So it's hard to tell which it is. But most likely it's the peering cross connect between Level 3 and AT&T on that OC 192

I would recomending do the test for longer and reposting it.

The last thign you want to happen is the typical:

You call Tech Support and show them the route

Tech Support who does fully under stand what they are looking at
a) Dismisses it yet promises some one will look into it.
b) Blinding sends it up to NOC as "Packet Loss" or "Problem here ..."

And when it gets to some one like me:

a) Ignor it as I can glance at it and see what the tech put in was wrong (Guilty of that) And no one calls you back. Or from then on blames it on you or something you are doing.

b) Belittle the tech and the supervisor who aproved that ticket and putting the fear of God in them when it comes to making tickets to me. (Guilty of that too)

c) Belittle the tech and supervisor yet catches that there may actually be a problem just not where any one says it was, and looks into it to see if it can be corrected or rerouted for lower latency, or rerouted till the problem is identified, confirmed and being corrected.


As you can see that is why some of people's complaints towards routing don't get adressed. But by large, alot of times there is no problem. Least not on the routing side.

[REP0MAN]

Thanks for the insight DerHelm.

My expertise and responsibility is the HFC network in our inner ring. Yes, Cox has some routing with Level3. Fortunately for me, had I needed to connect to a server in NJ, I didn't route through them and had a stable connection.

Hope you can get someone to look at your problem Yanks!

[yanksfan]

As you can see that is why some of people's complaints towards routing don't get adressed. But by large, alot of times there is no problem. Least not on the routing side.

OK, i'm not gonna pretend i understand all that, lol, i knew i should have run it longer however i didn't have alot of time at that moment, when i ran it earlier that one hop i pointed out was dropping between 14% to 60% not ten, this warping and lag has never been a problem for me until about 4 weeks ago.

rather then calling comcast,(we have had our battles) i'll give it a few weeks until i get around to trying a dsl, if thats better then i will dump comcast, they are a total pain in the azzzz and are very expensive anyway.

But I wanna thank you guys for taking the time to try and help me, it's really cool of you to do it.

Don

[Skuzzy]

Just FYI Derhelm.  PingPlot intentionally does not show packet loss down the entire line when it occurs.  It only shows packet loss at the first occurence then restarts.  That is according to what the Ping Plot guys have told me.

It is valid.  However, it would be better if the test was run longer.  But then again, it does not matter as packet loss in 10 seconds, versus packet loss in 10 minutes still kills the player's game connection to the server in terms of smooth game play.  Yes, if the packet loss consistent enought it will even kill the player's connection.

You can also run Ping Plot with UDP protocol as opposed to ICMP ECHO messages which would be a better indicator anyway.  Too many ISP's (AT&T& included) do block or have set ICMP ECHO messages to low priority.

If you do not mind me asking, which ISP do you work for Derhelm?

First level tech support (the first contact), at most large ISP's, will only deal with the last mile of the connection.  The folks I have talked to at a couple of them (do not ask me to name them) will not even acknowledge there is anything more than that involved with the Internet connection.  They merrily test the last mile and claim all is fine and that is all they are responsible for.

[DerHelm]

I tried out the packet loss test on pingplotter. It may be feature some where that I can't find, but by default if there is packet loss, it does carry on down the path.

I did this one with actual packet loss by killing my netscreen at home on my cable line (didn't want to kill my DSL line)



The packet loss here is cuased right at the first hop and passed all the way through.


Now this is what people generaly see. This is just a router that was either, too busy to handel the ICMP info in time, was secured against replying to TEMs or the lesser known too many ICMP proccesses (cisco default is 8 at any given moment)



Now hop 2 and 3 will not respond to ICMP at all. But it passes it through just fine. There is no actual packet loss on this route. These 2 routers are just overly secured. If they were just super busy instead or loss it would most likely show as latency and or loss even though there is no real data loss or latency.


The reason a longer run would be more valid is incase of either active or passive use of their internet connection. Packet loss dose happen usually on a very low percentage, but if that happend, losing 1 packet out of 10 tries shows 10 percent packet loss. Versus runnign for 30 mins.  Also vice versa, I've seen busting packet loss and latency that happens in chunks. So you may even miss seeing it.

Yeah, as you can tell I don't really specify who I work for. I get asked alot "What ISP should I go with" and my answer is usually "What are you looking for in an ISP and what do you do with your connection" And depending on the answer, some times is us, some times it's not. But I guess I do need some validation for all this being said. I am the network engineer and NOC serpervisor for DSL Extreme. We got bought abotu 2 years ago by IKANO, now we are IKANO Comunications dba DSL Exteme (DSL Mainly, T1, transit, hosting). Shortly after that purchase, I started having to take care of routing on Ikano's network (some DSL, dial up) then some other networks they own from other companies, Amerion (WA and OR) Some ISP in Canada, and Stic (some very small ISP in TX.) But don't judge me by those other network lol.

First level tech support is mainly there for last leg support becuase that's where most issues are. But if there is a problem past that they should be passing it on to people that take care of that.

It varies and who supports what not only ISP to ISP, but region to region and tech to tech (some need a kick in the butt for trying to convince customers everythign is their fault) on what can be done.

Some ISP's when it comes to routing there are different things to look at and depending on the issue.

If it's inefficient routing, that usually the hardest one if you are not multihoned. No bandwidth provider has the sortest path to every where. But having more Tier 1 providers and good peering in the right places help and gives you options.

The worst is the layer 3 re-sale lines, when I worked in Tech Support for Eathlink, all the Verizon DSL lines were just Verizon DSL, so getting anyting changed there wasn't going to be easy but it was possible. So as you can see it really varies. Unless we are talking about Bussiness class service, cuase if they did that to you wow did they mess up.

Side note on all this, even if there is a problem on another network (routing wise), you ISP has to take care of that. Lets say you are having trouble getting to some site. You go from AT&T to Level 3 then to the site and ther is a problem with a route on Level 3. AT&T has to look into rerouting around that problem and/or contacting Level 3 to find out why there is a problem and when it will get fixed. This is a standard practice for Teir1 to Teir 1 networks. Smaller networks are not worth is and you can eitehr force comunity strings or change you BGP tables.

[Skuzzy]

Certainly in that instance every hop will show loss.  You cannot simulate loss at one specific router along the path for any given period of time Ping Plot happen to hit it.

If the router burps, as it were, the packet loss will not show up in the rest of the path.

We see this happen all the time in the game.  One packet gets lost, but the others all make it.  Same principle involved.  PingPlot does simulate this condition pretty well.  I think you can best simulate this by using a filter to trap one packet in the middle of the stream and let the others all pass.

By the way, please check your PM's please.

[Getback]

How do I run a ping plot graph like the ones above?