There was a demo for a technology put out recently that circumvents this. I don't remember the exact mechanisms, but it obscured DNS such that your ISP couldn't see the DNS record you requested, and then used a proxy to route traffic before it hit the final endpoint eliminating exposing the IP to your ISP. It worked very similar to a VPN, but without the encrypted connection, and had some speed focused optimizations including the proxy being proximate to your ISP. It was pretty interesting.
They need the IP address to know where to forward the packet to. Hard to avoid that without VPN or TOR.
There was a demo for a technology put out recently that circumvents this. I don't remember the exact mechanisms, but it obscured DNS such that your ISP couldn't see the DNS record you requested, and then used a proxy to route traffic before it hit the final endpoint eliminating exposing the IP to your ISP. It worked very similar to a VPN, but without the encrypted connection, and had some speed focused optimizations including the proxy being proximate to your ISP. It was pretty interesting.