One way to look at it is loss of information. If you point a (brightness-tolerant) telescope at a lasar beam, you can't see much. Pretty much every beam is the same. If you point it at the sun there's all kinds of interesting features. Another way is to consider that light is a gas made of photons, and you're essentially talking about making every particle spontaneously align, which in a heat engine would obviously be ridiculous. All of these are entropy-negative processes, and a passive mirror or lens is passive and can only do reversible operations.

Another fun fact that comes out of this is that a magnifying glass can never make a spot hotter than the sun. Here's an XKCD what-if that goes into it - and might honestly be where I learned this first.