If you listen to more than one podcast, either

• you visit once a week anyway, and just have podcasts delayed a few days from release to listen, or
• you visit every day that one of the podcasts is released, which means you may be visiting several websites every day.

Some podcasts I like to listen to the day they come out, or perhaps the next day if I don't get to it, such as news podcasts.

Also, if you listen to even more than a few podcasts, you aren't going to "a website" once a week, you're going to a dozen websites once a week.

I just go to the website, download the show, throw it on my phone

That's three steps, per podcast per episode. Not everyone has their phone set up where it's zero-effort to copy files to the phone from their computer, so that may be a multi-step process itself.

Also, podcast apps offer some other features that to do manually either is more work, or more mental overhead:

• Favoriting episodes, so that they stay downloded: to do this manually you need some sort of filesystem hierarchy where you put favorited episodes, or keep a list of favorited episodes, or keep track some other way.
• Notifications for new episodes, for podcasts that don't follow a strict release schedule, or those that put out "special" episodes off their typical release schedule, or even just not having to memorize which podcasts have what release schedules.
• Viewing of "show notes" inline instead of having to open the browser, navigate to the podcast's webpage, then navigate to the episode page.
• Listening software designed for podcasts/human speech: silence trimming, speedup ratios, start/end trimming, smart chapter-based seeking and navigation, remembering where you left off. Some of these features may be available in whatever generic multimedia player you listen to podcasts in, but not all of them.

Of course, a podcast app is not required to listen to podcasts by any means. But if you listen to a lot of podcasts and value time your time, there is undeniable benefit offered by podcast apps.

Also, there are plenty of FOSS and tracker-free podcast apps, so it's not a situation where you must sacrifice privacy for convenience.