There are 3 servers, but they are all on the same IP subnet.
One hosts the special events arena.
Another hosts Early, Mid, and Late War arenas
The last hosts the Dualing, AvA, Training, and Late war arenas.
The big difference is the port range for each arena. If you are getting a host connection lost message on all arenas except the Dueling arena, that suggests somehting is blocking a pretty good range of ports.
Although if your computer is taking a very long time to preload textures, that could cause it as well. Uncheck any preload option in the "Video Settings" and try it again.