You was right with your first guess of lag (or better: latency) being the reason for this.
Most people connect lag invariably to warps, which isn't generally true. Signals do need time to travel the internet, so in a way we all have lag, some more, some less. Ping times vary from less than 20ms to >300ms depending on distance and type of connection. And of course the other player's and you own pingtimes, as the signals are transmitted from his computer to yours via the HTC server.
Warps are caused by variances in latency caused by bad internet connections oroverstrained computer resources.