There are currently 10 million vacant homes across the US, (15 million if you count the ones for rent,) which represents about 7.5% of the total housing. Not refuting your argument, just adding to the conversation.

I guess a lot of these are in less than desirable locations, among other factors, but 4.5 million of them are listed as seasonal, occasional, or recreational use. That's a lot of vacant housing, considering only 1.2 million of them are for sale.

Edit, I guess these are 2020 numbers so these may have changed since then.