It is not the last mile, per se. Although for cable users it could be, depending on how the local node is loaded.
It is mostly the gateway routers for the major ISP's, which is the problem. Large ISP's use regional gateways to help reduce the costs. The more bandwidth you buy, the less it costs per Mb/sec. Internally a large ISP will, usually, aggregate as much of the traffic they can into local hubs which then feed into the gateways.
The upside of the 'hub and spoke' design is it reduces the number of people to maintain it and it reduces the costs to the ISP for its bandwidth. The downside is, any group of customers, on any hub, could bring the network to its knees, for everyone, without throttling of some kind.
So when a Youtube or Netflix gets popular, it can reduce a well running network into a mess really quickly. Quite frankly, if I was running any of the large ISP's I would throttle the inbound traffic from Yourtube and Netflix in order to keep the network from being swamped for everyone. It is the only other alternative to raising prices and building out more infrastructure.
All ISP's over-subscribe. They have to. You want a one-to-one ratio? Are you prepared to pay a $1,000.00 a month for your Internet? The ratio has not changed for the ISP. The only thing that has changed is the amount of data per customer the ISP has to carry today.
I am glad I am not in that business anymore.