Any networking experts here?

[Bizman]

lol, I did not mean it in a bad way. I need ideas that that are out of the box since we looked and still looking inside the box. It was actually meant as a compliment.

I took it as a compliment in the first place. I've learned through this thread that you've got more than enough of expert level advice concerning your problem, that's why I dared offer my less stellar ideas. Hope you get it sorted in one way or another.

Besides, your words really were kind and I like being thanked as much as anyone else.

[dedalos]

I took it as a compliment in the first place. I've learned through this thread that you've got more than enough of expert level advice concerning your problem, that's why I dared offer my less stellar ideas. Hope you get it sorted in one way or another.

Besides, your words really were kind and I like being thanked as much as anyone else.

lol. Good. Most ideas will be wrong but one of them may turn on the light. You never know. 

[Vulcan]

Why I post it here, is that the cable modem is made by Cisco who are a better known brand in professional equipment. Household gear may be smaller and cheaper, but they have to provide equal protection against networms and such as their professional versions. The function in question was ip-flood-detection, disabling of which solved my client's problem. Wouldn't be the first time some safety feature causes trouble...

FWIW Cisco cable modems are usually just linksys gear rebadged, and Cisco security products are awful (yes lots of people buy ASA's, but lots of people buy Big Mac's too).

dedalo's the easiest thing to do would be to put a box in between the switch and the server/device missing packets, and drag through packet by packet. But BoilerDown hit the nail on the head, UDP is a stateless protocol, combined with the nuances of multicast whoever set the system up needs a kick up the backside. I would put money on it being related to membership and timeout requests (the switch needs to maintain and refresh the list of devices partaking in the multicast session). There are going to be tiny periods when 'administrative' tables are updated, and that could lose you packets.

If the data is so important it should have been on tcp or the app should have it's own embedded error checking for the UDP data.

[dedalos]

dedalo's the easiest thing to do would be to put a box in between the switch and the server/device missing packets, and drag through packet by packet.

A little more info on where you are going with this please.  Lets say I did this and we still drop, what would it point at?  The problem is all the hardware we own is exactly the same.  Lets say I don;t see any drops, what do I check yet?

But BoilerDown hit the nail on the head, UDP is a stateless protocol, combined with the nuances of multicast whoever set the system up needs a kick up the backside.

lol, yes we are familiar with networking 101 but I will put a call in to the exchanges to explain that they need to use TCP instead of multicast.

I would put money on it being related to membership and timeout requests (the switch needs to maintain and refresh the list of devices partaking in the multicast session). There are going to be tiny periods when 'administrative' tables are updated, and that could lose you packets.

Again, a little more info please.  In 15 years I have not witnessed this behavior.  The server will only drop when the 10Gig line is below 2 or 3% utilization but packets are coming in in the thousands per second even at those levels.  Why would the memberships time out and how do I tell if that is what is happening?

If the data is so important it should have been on tcp or the app should have it's own embedded error checking for the UDP data.

It does, but it does not explain why the packets drop, does it?

[katanaso]

If this was my network, I would be debugging the routers at the various interfaces.  I would also have a sniffer set between devices, changing it once I knew that the traffic was making it to that next router or switch.

I would doubt this is on your computer(s), and since your network is managed by somebody else, you may need your team to get on them about debugging their routers and switches, however the topology is designed.

Do you even have access to the last router on your side, or is that still managed by another company?

[dedalos]

If this was my network, I would be debugging the routers at the various interfaces.  I would also have a sniffer set between devices, changing it once I knew that the traffic was making it to that next router or switch.

I would doubt this is on your computer(s), and since your network is managed by somebody else, you may need your team to get on them about debugging their routers and switches, however the topology is designed.

Do you even have access to the last router on your side, or is that still managed by another company?

If we are talking about physical access, no unless I go to the data center.  I could if I needed to but I would need to know what I am looking for.  Can anything of what you mentioned/thinking be checked remotely from my servers?  There are three parties involved in this.  Me, the network host and the data provider.  I am willing to take their word that they have checked their end and don;t see the drops.  Do I trust them 100%?  No, but I can only push so much without any proof and they are risking legal action if they are lying. I don;t have access to the data providers equipment. My hope is that it is something on my end because I can fix that once I figure out what it is.  If the drops happened during high utilization it would be easy to come up with ideas.  But it doesn't.  Do you know of any settings on a switch that could cause this?  Anything would be helpful no matter how crazy it sounds.  I can get the equipment specs on Monday if you think they could help.

[katanaso]

You wouldn't need physical access.  Sorry, I should've been more specific.  I just meant access to the configuration of the routers.

There could be a number of reasons why it's happening, but on each router, I'd look at the error logs for the port and see what's there.  It would give you an idea of where to start.  If you're seeing lost packets, or other signs of network issues on the physical device, you can troubleshoot from there.  If the logs are clean, and you're not seeing issues, move on to the next router or switch.

There are various possibilities on layer 2 and layer 3.  Just an example from a Cisco site:  http://www.cisco.com/en/US/tech/tk828/technologies_tech_note09186a0080094b55.shtml

You can install a free packet sniffer on your servers as well, such as Wireshark, and analyze those logs.  However, that's where physical access might come in handy by plugging a sniffer (or laptop with a packet sniffer) into the same physical device to ensure you're capturing all of the traffic.

I would ask the network host to do some debugging.  I don't believe it would be on your data provider's side, at least not at first.  It could be something regarding MTU sizes.  An example from Juniper:  http://kb.juniper.net/InfoCenter/index?page=content&id=KB25312

So you know, I am out of 'hands on' for several years now, and I have a staff that would handle these problems, but in managing it forward, I'm just relaying how I would do it on my network, and who I would have my staff contact.  I'd absolutely want to debug the interfaces and go from there.

[dedalos]

You wouldn't need physical access.  Sorry, I should've been more specific.  I just meant access to the configuration of the routers.

There could be a number of reasons why it's happening, but on each router, I'd look at the error logs for the port and see what's there.  It would give you an idea of where to start.  If you're seeing lost packets, or other signs of network issues on the physical device, you can troubleshoot from there.  If the logs are clean, and you're not seeing issues, move on to the next router or switch.

There are various possibilities on layer 2 and layer 3.  Just an example from a Cisco site:  http://www.cisco.com/en/US/tech/tk828/technologies_tech_note09186a0080094b55.shtml

You can install a free packet sniffer on your servers as well, such as Wireshark, and analyze those logs.  However, that's where physical access might come in handy by plugging a sniffer (or laptop with a packet sniffer) into the same physical device to ensure you're capturing all of the traffic.

I would ask the network host to do some debugging.  I don't believe it would be on your data provider's side, at least not at first.  It could be something regarding MTU sizes.  An example from Juniper:  http://kb.juniper.net/InfoCenter/index?page=content&id=KB25312

So you know, I am out of 'hands on' for several years now, and I have a staff that would handle these problems, but in managing it forward, I'm just relaying how I would do it on my network, and who I would have my staff contact.  I'd absolutely want to debug the interfaces and go from there.

Cool, thank you.  I have remote access to the servers and they all have wireshark installed.  I have to find a way to use it without impacting production since the amount of data coming in is ridiculous lol.  I would have to ask the admins for the logs since I don't have access to the network.  However, they do claim they are clean.  Time to look for my self I guess.

Thank you

 

[Bizman]

FWIW Cisco cable modems are usually just linksys gear rebadged, and Cisco security products are awful

I've read about Cisco owning Linksys, my point was merely a guess that security related solutions might be similar in all size systems.

@dedalos: Yet another crazy idea: Is it possible that your network host has got new gear, which might be slightly incompatible with yours? I used to suffer from Internet problems twice, when my ISP refurnished. The first time they had to update the firmware of their new gadgets, the second time I thought getting a new modem would be the most viable solution.

[katanaso]

Cool, thank you.  I have remote access to the servers and they all have wireshark installed.  I have to find a way to use it without impacting production since the amount of data coming in is ridiculous lol.  I would have to ask the admins for the logs since I don't have access to the network.  However, they do claim they are clean.  Time to look for my self I guess.

Thank you

 

Not a problem at all.  It would take a few minutes for you or the network admins to log into each device and get a status on the interfaces that are used.  Can you sit next to the network admins while they troubleshoot?

[dedalos]

Not a problem at all.  It would take a few minutes for you or the network admins to log into each device and get a status on the interfaces that are used.  Can you sit next to the network admins while they troubleshoot?

The equipment is in Chicago, the admins in NY (although they have staff that can go into the data center if they need to) and I am in Texas lol

[katanaso]

The equipment is in Chicago, the admins in NY (although they have staff that can go into the data center if they need to) and I am in Texas lol

That is funny.

Perhaps ask if you can view their desktop while they debug, so you can ask questions in real time.

[Vulcan]

A little more info on where you are going with this please.  Lets say I did this and we still drop, what would it point at?  The problem is all the hardware we own is exactly the same.  Lets say I don;t see any drops, what do I check yet?
lol, yes we are familiar with networking 101 but I will put a call in to the exchanges to explain that they need to use TCP instead of multicast.

Again, a little more info please.  In 15 years I have not witnessed this behavior.  The server will only drop when the 10Gig line is below 2 or 3% utilization but packets are coming in in the thousands per second even at those levels.  Why would the memberships time out and how do I tell if that is what is happening?
It does, but it does not explain why the packets drop, does it?

Well, you have unicast broadcast and multicast right. Unitcast is point to point right where the send specifies the address of the destination. Broadcast goes to everyone. For multicast you have to tell the switch you're connected too that you want too partake in a multicast session, that switch in turn may need to tell upstream devices. IIRC the multicast table refreshes every 60 - 120 seconds. Hell in low traffic scenarios where there is no activitiy it could even be something as simple or weird as an ARP timeout.

[dedalos]

Well, you have unicast broadcast and multicast right. Unitcast is point to point right where the send specifies the address of the destination. Broadcast goes to everyone. For multicast you have to tell the switch you're connected too that you want too partake in a multicast session, that switch in turn may need to tell upstream devices. IIRC the multicast table refreshes every 60 - 120 seconds. Hell in low traffic scenarios where there is no activitiy it could even be something as simple or weird as an ARP timeout.

Its worth taking a look at although low traffic for me is probably high traffic for other systems but it may give the network guys some ideas.  Thank you

[eagl]

Possibly a wild goose chase, but is there a chance that an upstream host is traffic shaping you to another pipe when your bandwidth in use drops below a set level?