I don't really want to use a USB TPU (no real reason)
Since there is no real reason, why no reevaluate?
I connected my USB TPU to a spare internal USB 3.0 header and mounted it inside the case. Frigate has been working perfectly fine for years.
I also use quite a few PCIe M.2 adapters but only for SSDs. No complaints here, full transfer speed and reliable. No reason why it shouldn't work for TPUs.