I dont give anyone that number except for the purpose of 2FA, so SIM swapping wont work, the sim card never leaves the house, so scan based exploits wont work, and the phone doesnt have the hardware required to be vulnerable to more sophisticated phone based attacks.

I understand what you're saying. But the reality is everything you just said doesn't matter for SIM swapping. The fact that you use the phone number for that service says that the number you use is out in the wild. Typically when SIM swapping is used is when there's a data breach and your username, email, password, and phone number are leaked. But they still can't get in because of the extra 2FA step.

So they HAVE that phone number. SIM swapping is done at the carrier level. It's when the associated number is "swapped" to a different SIM card (one that the hackers own). Which means you can get totally screwed over without lifting a finger and not a single person touching your computer or phone.

Like I said before, the damage to you would be done before you even knew what happened.

Edit: autocorrect

Edit 2: and yes, I understand many services have no other options than SMS, which is why it's such a huge massive problem.