Even if he is accurate, specialist hardware will outperform generic hardware at what it is specialized for.

I remember a story sometime in the 00s about PCs finally getting to the point where they were as fast as one of the WWII code breaking computers (or something like that). It wasn't because we backtracked in computer speeds after WWII, but because even that ancient hardware was able to get good performance when it was purpose-built, but it couldn't do anything else and likely would have required a lot of work to adjust to a different kind of cypher scheme, if it could be adapted at all.

So GP compute might be a million times faster in a decade, but specialist AI chips might be a million times faster than that.

A hardware neural net might be able to eliminate memory latency by giving each neuron fast resisters to handle all their memory needs. If it doesn't need to change connections, each connection could be hard wired. A GPU wouldn't have a chance at keeping up no matter how wide that memory bus gets or how many channels it gets split into. It might even use way less power (though with the elimination of memory latency, it could go fast enough to use way more, too).

Nobody is saying it won't happen eventually. But a million times within the next decade, i.e. 4x better every year for 10 years?

This generation isn't better than last generation by even close to that. Nevermind doing 4x for 10 years straight.