There are a lot of competing factors here, and also a consideration for what we consider beautiful.

Without getting too wanky about it, I'd guess that what most people consider beautiful is your "traditional" village with two storey detached houses and mature gardens. A little parade of local shops, a pub, green, a pre-reformation church, etc. Not many of those exist, and those that do are either insanely expensive, out in the middle of nowhere, or both.

Then you've got density. Because the UK is so incredibly centered around the south east - due to policy failures over literal centuries, not just decades - you've got to build up. The problem with "skyscrapers" is that they are not the 'street in the sky' they were promoted as when council tower blocks were all the rage in the 60s. For one, there is a complete absence of public shared space. Corridors lit with bulbs are not streets in the summer twilight. That shared space is really what that village above is all about, yes you have your nice house, but the rest - the community you're also part of - only comes about by being essentially forced to interact with each other.

If you've never lived in an apartment building it's really difficult to describe just how isolated you are. You never even see your neighbours, because you don't get that happenstance of looking out the window watching the birds and notice the person across the road getting home, the same way you do in other forms of housing. The only chance you get is if both of you just so happen to be going down the same narrow corridor at the same time. There is no community because there is no familiarity. Because there is no community, everyone is faceless, so "anti social behaviour" doesn't matter cos instead of it being Fred who you are disturbing at 2am, it's "that prick next door who always complains".

Balancing density and maintaining communities is a hard problem; as you say, it takes much more than just trying to cram every m^2 of land with directly sellable property.

There is a balance between individualism and community. We've seemed to develop a view of individualism which is about everyone being an island and not needing anyone else, rather than about the freedom to be true to yourself, and tolerant of others, whilst recognising we are all humans, and ultimately require each other to not just survive, but thrive.

The best example I can think of this was when I was speaking to a friend the other week. They are buying a house and one of their conditions was a big enough garden so their kids can play football. Not a full size pitch, obviously, but a good 40-50 meters.

When I was a kid, we did that on the community playing field a 10 minute walk the house, on a full-size pitch, with sticks in the ground (the council later installed goals). The estate they were looking at, a 10 minute walk meant you were still in the estate. It's a new build, has a tiny soft play area, and nothing else. It's going to be a warzone in 10-15 years when all the kids have grown up and have absolutely nothing to do.

Edit: this got away from me slightly...