Any networking experts here?

[dedalos]

Does anyone know why I would be seeing packet loss on a multicast stream only when the utilization is low?  For example, I have a 10Gig connection that pulls in market data.  At 1% utilization I see packet loss every few seconds.  At 3% and above I see none.  As long as the network utilization stays above 3% or 4% I don;t see a single drop.  Needless to say we have tried every possible setting we and the nic maker could think of with no change on the results.  Any ideas?  I'd rather not work of the machine specs since I am looking for ideas on what could be causing something like that in general but if you need the specs I ll post them.

Thank you

[Bizman]

I'm no networking expert, but one thing that comes into mind is that Windows might be shutting your network card down if there's no traffic. Checking the Power Management settings for your nic and unticking the "Let windows shut this down" would be the first step, which you probably already have done with the nic maker. Also, if you have tinkered with Windows Services, could it be possible that you've set some network related service from automatic to manual?

[Trell]

I am not sure how much of the network is yours.  Are you seeing this  on the end server or are you seeing the packet loss on the router connected to your providers wan?

When I used to do more technical work I dealt more on the VoIP side of things. But normally I would put a sniffer as close to the entry point this data is coming into your network or as close to the origination server of this data to make sure it is not an issue father up stream.

[dedalos]

Hey Biz, yes all that stuff has been checked. 
Trell, unfortunately the network is managed by a different company.  They have put sniffers and checked the equipment and they claim to see nothing wrong.  Everyone thinks it is at the server (receiving end) but I am open to anything really.  I am hoping that someone has seen this before and figured it out in the past.

I am looking for the crazy out of the box idea here since we have checked everything we could think of with several people.  I think the key is that it only drops when there is low utilization on the line.  This is market data so utilization is never at 0%.  As long as I can stay above 3 or 4% utilization I don;t see any drops.  The higher the better lol. 

[Bizman]

Is there only one or more computers in your end? If only one, how about another network card of another brand just for testing purposes?

[Trell]

If you have any control of your own switches you could set up a mirror port and set up a laptop to run a sniffer on just to be sure. Depending on traffic through put you could put a hub in front of the box to do the same thing.
I would second changing out the nic card was well.  It is a simple thing to do as a test..  normally when i see packet loss it is on the router or provider side,  even if they say it is fine....    Not that i don't trust managed networks but I don't trust Managed networks.

[dedalos]

Is there only one or more computers in your end? If only one, how about another network card of another brand just for testing purposes?

three servers (same specs) listening on 12 mc channels each. They all show the same behavior at different times.

[dedalos]

If you have any control of your own switches you could set up a mirror port and set up a laptop to run a sniffer on just to be sure. Depending on traffic through put you could put a hub in front of the box to do the same thing.
I would second changing out the nic card was well.  It is a simple thing to do as a test..  normally when i see packet loss it is on the router or provider side,  even if they say it is fine....    Not that i don't trust managed networks but I don't trust Managed networks.

Heh, no control what so ever over the network other than calling the network people.

[eagl]

Is QoS buffering packets in the hopes that something better will come along?

[Copprhed]

Dedalos, look on Facebook for "Cameraonastick". He's a former friend of mine and is a god in puters and networking. If he can't help you(and it will cost, it's how he makes his living) I truly don't believe anyone can fix it.

[dedalos]

Is QoS buffering packets in the hopes that something better will come along?

Well, it should be off.  Is it possible the Windows is ignoring the setting somehow? 

[DubiousKB]

Think physical... Perhaps there's a Electromagnetic Field or Radio Frequencies near your equipment that only induces the "noise" when the line is inactive. When utilization begins to ramp up, the minor EMI, RFI fields are negated and you do not see packet loss....

Turn off any HVAC & non-essential Electrical systems and see if you notice a difference.....Just a theory from my experience...  Had an industrial Air Conditioning unit throw a wicked EMI field around some local wireless networks and found it to be intermittent as hell until we realized the physical proximity of the motors and our access point.....

Hope that helps and not hinders!

[Trell]

Here is a question i should have asked before.  How are you determining that you are getting packet loss?  Is it from the application that is receiving the data  or is it a different 3rd party monitoring system that is showing it?  I am also wondering what kind of % packet loss you are recieving

[dedalos]

Sequence numbers from the app, netstat, wireshark

[dedalos]

Think physical... Perhaps there's a Electromagnetic Field or Radio Frequencies near your equipment that only induces the "noise" when the line is inactive. When utilization begins to ramp up, the minor EMI, RFI fields are negated and you do not see packet loss....

Turn off any HVAC & non-essential Electrical systems and see if you notice a difference.....Just a theory from my experience...  Had an industrial Air Conditioning unit throw a wicked EMI field around some local wireless networks and found it to be intermittent as hell until we realized the physical proximity of the motors and our access point.....

Hope that helps and not hinders!

lol. Kind of what I m looking for since we have tried the conventional aproaches.