Just to clarify what I'm experiencing.
I don't believe the over the air signal is the problem. Both my phone, and the mobile hotspot I got show a full strength 4G LTE signal, which never waivers. But in spite of the strong signal the internet is slow with horrible latency/variance.
I'm led to conclude therefore that it's not the signal from me to the tower that's the issue. But perhaps whatever connection the tower uses to get to the internet.
On the same network in town, with less bars (signal strength) on my phone, and no LTE (just plain 4G) I tested and get 100-150ms ping and 20-25 mbps download. vs at home I get 300-700+ ping and 0.2 to 4 mbps download with a perfect 4G LTE signal. It varies a lot too, tonight it seems to be marginally better then it was last night.
So thanks for the suggestions, but I don't think a signal booster will make a difference. I wonder if there is just more data traffic to the tower than it can handle...
EDIT: I'm gonna run pingplotter and post the results. Maybe there will be some clue there, I just don't know how to interpret it.