We instead grow large amounts of crops that go to animal feed. It takes a lot less cropland for plant-based diets because we don't have to grow feed to another creature (who then will use up a large amount of that energy)

The research suggests that it’s possible to feed everyone in the world a nutritious diet on existing croplands, but only if we saw a widespread shift towards plant-based diets.

If we would shift towards a more plant-based diet we don’t only need less agricultural land overall, we also need less cropland.

In the hypothetical scenario in which the entire world adopted a vegan diet the researchers estimate that our total agricultural land use would shrink from 4.1 billion hectares to 1 billion hectares. A reduction of 75%. That’s equal to an area the size of North America and Brazil combined.

https://ourworldindata.org/land-use-diets

I think you may be underestimating the heavy level of subsidies here

Research from 2015 shows this subsidization reduces the price of Big Macs from $13 to $5 and the price of a pound of hamburger meat from $30 to the $5 we see today.

https://www.aier.org/article/the-true-cost-of-a-hamburger/

Even despite that, overall in most countries it actually ends up being cheaper to do a healthy plant-based diet assuming you have more whole-foods and less say plant-based meats

It found that in high-income countries:

• Vegan diets were the most affordable and reduced food costs by up to one third.

• Vegetarian diets were a close second.

• Flexitarian diets with low amounts of meat and dairy reduced costs by 14%.

• By contrast, pescatarian diets increased costs by up to 2%.

https://www.ox.ac.uk/news/2021-11-11-sustainable-eating-cheaper-and-healthier-oxford-study

And real world data backs this up

Compared to meat eaters, results show that “true” vegetarians do indeed report lower food expenditures

https://www.sciencedirect.com/science/article/abs/pii/S0921800915301488?via%3Dihub ---(looking at the US)

Based on primary data (n = 1040) collected through an online survey, representative of the Portuguese population, through logistic regressions, it was possible to conclude that plant-based consumers, particularly vegan, are associated with lower food expenditures compared to omnivorous consumers. In fact, plant-based consumers are shown to spend less than all other consumers assessed

https://agrifoodecon.springeropen.com/articles/10.1186/s40100-022-00224-9