It's still resources that could be spent towards something else, something ultimately more productive.
Building a house takes a lot of work, so why spend that effort into building a bomb that destroys many such houses, instead? What does this achieve for humanity?
Speed limits are not bs. The energy of kinetic motion increases with the square of the velocity. Double the speed, quadruple the amount of energy. That matters.