or indeed hundreds of great selling AAA single player games?

It's important to note that the amount of single player AAA games has greatly diminished overtime. Most of those "hundreds" you're referring to are not in the last 10 years, and the big bucks have been in live service. So yeah BG3 did great but it was a huge, 6+ year gamble ultimately. I WANT those gambles, but businesses would rather push out cheaper games at a faster clip because they make money. People still buy them and they still pay for DLC/MTX like crazy. It's hard to compete against that.