QoS generally sucks and can slow everything down if your router itself is too slow. But if you're going to do it, you want to set their connections with the limitations, and leave yours unlimited and highest priority.
Wireless sucks more though, its half-duplex and subject to collisions from every other wireless communication on overlapping channels in reception range, even your neighbors across the street that don't have access to your router. By far the best option is to string a cable from your router to your computer. Even a temporary 300 foot cable that you only connect while you're playing could solve your issues.