Yes it can be done on a budget, although the more you get into it, the more alluring the expensive stuff is.

Even without a license, you can get started with some listening only stuff. For me, I picked up an rtl-sdr off Amazon for pretty cheap and started listening to NOAA weather satellites.

Before you start transmitting, you'll need to get your license. Hamstudy.org is a great place to start practicing for the exam.

Study for your tech license with https://ham.study/ and learn how to use a baofeng uv5r/gt5r handheld. That's how I started on a budget. Don't overspend on the baofengs on amazon. They should realistically be around $25 not $60 and they are all almost the same.

Definitely can be done on a budget. As others mentioned, you do need a license, but study materials are free. The Technician license is the most basic.

Generally there will be an amateur radio club around most areas -- I'd do a search for your area. If there is one near you, and especially if they hold physical meetings, just get involved!

An SDR (software defined radio) is likely the cheapest way that you can start listening (not transmitting), and that can be helpful to hear how people communicate. Again, do a search and see if there are radio nets local to your area. The most basic antenna is a long strand of wire!

And finally, just a heads up: ham, not HAM. Contrary to popular belief, it isn't an acronym for anything.