Video is a can of worms. Video files have the concept of a container (like .mp4, .mkv) and codecs (h.264, h.265). Making it worse, you also have embedded audio which has its own codec (mp3, ac3, aac, ogg, flac).

For me, mp4 container with h.264 video and aac audio have the widest hardware support- videos encoded that way pretty much play on everything and well.

It’s going to be a couple more years before good hardware support for h.265 is ubiquitous. I see .mkv with h.265 used a lot for 4K stuff, and while well supported on desktop, device support is still spotty.

Video is tougher as it changes every few years. Once a new standard is out there is also a little lag before there is hardware decoding. Additionally the video codec and extension are different things. I’m going to talk about codecs as I think it’s more important than file type.

Hardware decoding is important especially for mobile devices and software decoding will run the processor and eat up your battery.

For video, right now, the technically best format I think is AV1. If I’m not mistaken it’s open source or at least doesn’t have royalties.

Images are kind of the same story, but browser support is the issue.

I think jpegxl is supported to be pretty great, but Google is trying to push their own standard which I believe is technically inferior in some ways. Because of that Google had support for jpegxl in chrome and then they removed it from chrome.