To some extent, that's true. But getting the data off of your phone is the first step. And that is where you have the most control and the bottleneck of poor internet service and data caps prevents transmitting too much data for now.

Audio data can only be compressed so far before it becomes impossible for a server to transcribe it. And you're talking about a constant stream of background audio which means you can't afford to lose much of that data to compression at all. The device might be able to differentiate speech from background noise and only send the stream to the server when someone is speaking, but that's about it for the large majority of devices.

To be able to interpret all speech, including accents, takes a server still, unless it's a high end device and you don't mind battery drain making the user suspicious. It's just not feasible with current processor, battery, and bandwidth limitations to listen to everything.