Radiant heat transfer in the real world often appears quite odd in its behaviour despite being seemingly simple. I learned quite a bit about it when I decided to implement radiant ceilings in my home.

Yes, the panel radiates the same amount regardless of where it's pointing. However, other rays are incident on it from other surfaces that deliver heat back to the surface. Thus the point of a selective emitter that emits more than it absorbs. Likewise solar thermal panels are optimally made from selective absorbers, but IRL flat black paints are so much cheaper that it's not worth it.

So thermal comfort often is a result of radiant balance. Your 20° clothing radiates to the walls - the 20° walls radiate to you - there is no net loss of heat, and you are warm.

Step out under the dry, cloudless prairie sky at night, you radiate into the infinite blackness of space. Nothing radiates back. You cool off rapidly. It's not so much that the heat needs to be dumped into space, but that space offers no heat in return.

Seriously it's pretty neat to point my thermal scanner at the night sky and see it read -INF. The night sky is an effectively unlimited radiant sink.