Packet loss is indicative of routers which are simply overcrowded or a connection which is saturated.
Servers are typically connected via dedicated Ethernet connections so they are far less prone to dropping packets than routers.
Ethernet switches can also drop packets, but are less prone to do so than routers, but more prone than servers.
If the connection you are seeing packet loss on is at your ISP's gateway to another ISP, then you can contact your ISP about it. However, be aware they probably will not do much if the routers are not having an issue outside of overcrowding.
But, you never know until you try.