If your services are not stateless, work to make them such so you can learn about scaling in the cloud, which can even be done w/ VM-based services. how much more agility using cloud vs a DC gives you

This can't be understated. Embracing elastic idology to remove single points of failure and decoupling stateful aspects of applications has been the biggest takeaway of being part of several migrations of services to AWS. Implementing these into your practices as you grow is a huge benefit that may is worth the cost.

Over time, if the scale you're operating at grows, using experience/knowledge from AWS and applying it to running services in a datacenter could be beneficial. In my experience, if you have a large, consistent, asynchronous workload which you've maxed out on reserved instances or savings plans, it is likely cheaper to operate on your own hardware than in the cloud (or get credits from GCP or Azure to migrate services to reduce costs). This is where avoiding vendor lock-in is key.

have y’all factored in all the time/money spent on maintaining the server hardware, power, DC cooling, etc. too?

For sure, this isn't 2007 where you need to purchase servers and network equipment to start a website. For most startups and small businesses, operating in the cloud will be less expensive upfront and likely over the first 3 years. This isn't a one size fits all approach though, and it'd be prudent to evaluate the cloud spend periodically and compare with what'd it'd cost to manage it entirely. Obviously you'd need a team competent enough to manage this, without it going to shit.