Any networking experts here?

[dedalos]

Possibly a wild goose chase, but is there a chance that an upstream host is traffic shaping you to another pipe when your bandwidth in use drops below a set level?

Everything is possible I guess.  It should be a direct connection from one data-center to the other but . . .

[eagl]

Everything is possible I guess.  It should be a direct connection from one data-center to the other but . . .

Simple route tracing over time might work unless they're using virtual servers or routers or just throttling without changing IP addy.

Or, maybe when bandwidth drops some automated housekeeping starts which sucks cpu time resulting in dropped packets?  Like the idiots who assume everyone goes to lunch at noon so they force start the corporate antivirus scans at noon bringing the entire IT infrastructure to its knees for an hour as every hard drive on the network starts thrashing?

[dedalos]

Simple route tracing over time might work unless they're using virtual servers or routers or just throttling without changing IP addy.

Or, maybe when bandwidth drops some automated housekeeping starts which sucks cpu time resulting in dropped packets?  Like the idiots who assume everyone goes to lunch at noon so they force start the corporate antivirus scans at noon bringing the entire IT infrastructure to its knees for an hour as every hard drive on the network starts thrashing?

To be honest, I think that is what is happening.  Not a person but the servers or the nics are ignoring some setting or something else is telling the OS to turn them back on.  This is the card I am using: Solarstorm SFN5162F SFP+ Server Adapter.  When this thing gets bz it runs flawlessly so I know the hardware can take the load.  Something is putting something to sleep, but what or how do I find it if when looking at the settings everything has been turned off?

[eagl]

Still strange.

My old laptop had pci-e bus power savings options, but the settings were visible only from the ATI video card drivers.  There may be a data bus power savings mode kicking in.  That could be pretty deep in the drivers.  Might need to get some serious OS support going if you need custom drivers.  What OS?  Do the machines have a BIOS you can access, to try turning off all power savings settings?

[katanaso]

Along eagl's lines - what OS are the servers running, and what make and model are the servers?

[dedalos]

Windows Server 2008 R2 Standard SP1 dual Xeon 2.3Ghz

[Bino]

Think physical... Perhaps there's a Electromagnetic Field or Radio Frequencies near your equipment that only induces the "noise" when the line is inactive. When utilization begins to ramp up, the minor EMI, RFI fields are negated and you do not see packet loss....

Turn off any HVAC & non-essential Electrical systems and see if you notice a difference.....Just a theory from my experience...  Had an industrial Air Conditioning unit throw a wicked EMI field around some local wireless networks and found it to be intermittent as hell until we realized the physical proximity of the motors and our access point.....

Hope that helps and not hinders!

One of our offices recently had a problem with comms between a computer and a check scanner that was ultimately traced to a nearby powered stapler.  D'oh!  <facepalm!>

[leitwolf]

Dedalos, another angle could be: if you mix 1G and 10G Ethernet (say your servers have 1G interfaces, and the distribution switches use 10GE) this can produce packet loss by even the most modest spikes in traffic.

(There is a long winded explanation of how this can happen, it needs a speed mismatch between some of the interfaces in your traffic flow - if that's not your setup disregard this post - if it is i can explain it )

[dedalos]

Dedalos, another angle could be: if you mix 1G and 10G Ethernet (say your servers have 1G interfaces, and the distribution switches use 10GE) this can produce packet loss by even the most modest spikes in traffic.

(There is a long winded explanation of how this can happen, it needs a speed mismatch between some of the interfaces in your traffic flow - if that's not your setup disregard this post - if it is i can explain it )

its the opposite remember?  High load = no problems.

[leitwolf]

ah, my fault, only read half of your post apparently

[dedalos]

Lol. Although, there is a 1gig interface also. 10gig input and 1 gig out put on different nics.

[eagl]

Disable any kind of jumbo frames   Maybe some nics are combining packets when they have the processing time overhead and there is a compatibility issue somewhere.

Best way on my home network to induce instability is to enable jumbo frames on everything.

[dedalos]

Disable any kind of jumbo frames   Maybe some nics are combining packets when they have the processing time overhead and there is a compatibility issue somewhere.

Best way on my home network to induce instability is to enable jumbo frames on everything.

no jumbo frames used or allowed, ever lol.

[dedalos]

I found this http://support.microsoft.com/kb/2639824 I will install it and see what happens.  Not holding my breath though lol.

[Trell]

Silly question.  Have you opened up a ticket with Microsoft on this issue yet?  their support is normally good.  We have went to them on some weird problems before.  The cost is next to nothing.