Apart from the philosophical aspect which is unanswerable, I feel certain we see equivalent colors. This is an interesting article about it. https://en.wikipedia.org/wiki/CIE_1931_color_space Scientists found what the three primary colors our eyes see are. Because of the overlap in cone activation they're actually imaginary colors that don't exist.

I'm red-green colourblind(Deuteranopia) and often think this exact thing, how the reality I perceive is different from others purely due to this.

I can't describe the difference of red vs blue and I've never met anyone else who could either.

In the Mask movie there is a great scene where he demonstrates colors to his blind girlfriend. They did a really great job with it.

I've had this thought many times, glad I'm not alone. Also makes you wonder if possibly everyone's "favorite" color is the same color, we just all call it different things because of how we individually perceive it.

This is a fun thought, but I can disprove this myself easily enough due to having had my favorite color change multiple times in my lifetime. Currently enjoying green.

Scientific research indicates we see colors pretty damn similarly, with edge cases for colorblindness and also people who are more color sensitive.

One way this can be studied is by studying the metamerism of different colors by different observers. Metamerism is the study of how colors change given different light sources.

There are other objective qualities that give hints that we have similar ways of experiencing colors. You mention that colors are nothing more than our brain assigning "color" to frequency of light -- but light is itself just a frequency of electromagnetic radiation, namely the frequencies that make up the bulk of the radiation emitted by the sun.

So to a normal observer without colorblindness, there are more variants of colors of green than any other color. Green is of course situated in the very center of the roygbiv spectrum, it is the "most visible" color. The colors with the least amount of variations are red and violet, which are situated at the edges. Frequencies above violet or below red become invisible making up infrared and ultraviolet radiation.

Where we get tricked up, and I used to have identical suspicions as you did, is that we consider color to be purely subjective, because we aren't taught to unify subjectivity and objectivity into a united whole. Color isn't completely imagined, there are certain surfaces that absorb and reflect certain frequencies of EM radiation just as the structures in our brain that process this ocular input are more or less similar. Things that are subjective aren't usually associated with being "real" the same way that objectively "real" things that exist out in the phenomenal world are. However, color is socially real, we can almost all identify colors that are the same and colors which are different. Since the set of colors which are "red" are fewer than the set of colors which are "green" then there is no way that what I experience as red is the same as what you experience as green. Artists use colors to convey emotion and are able to achieve this with many many different observers. Warm colors are warm, and cool colors are cool. There may be different levels of sensitivity but in my experience this can be somewhat trained into an observer though no doubt there are outliers who have a unique sensitivity to color differences.

So there are objective factors which align with subjective factors let's say 90% of the time, which strongly supports the idea that we experience color more or less the same way. The trouble is not that subjectivity and objectivity are irreconcilable, in fact it is when we fail to reconcile them that our troubles begin. In my opinion, this is a huge problem that creates all kinds of issues when we try to relate to each other; it may be the most prominent philosophical problem of our age. Luckily it is fairly easily remedied with a slight change in the way we think about subject and object. Its useful to separate them sometimes but we need to be able to reunify them, which just takes practice in my experience.

Well, it really makes sense for these every specifically tuned biological machines to all function more or less the same way.

Everything we can glean from neurology pretty much says our perceptions are similar, we just process them differently.

Red is a shorter wavelength than blue. It would make no sense for the brain to interpret long wavelengths as short or short as long, which is probably why our colour perceptions are more or less the same.

Language affects our perception more than the biological hardware we have. The physical sensations are similar to everyone, but processing them is different. Which is why it could still be that your red isn't my red. But my point is I don't think it'd ever be blue or green in any context. It'd je different, perhaps, but not fundamentally so.

The ancient Greeks used to call the sky bronze. Related, there was this cool short the other day. Talked about how someone raised their kid normally other than carefully making sure never to say what colour the sky is, and then later inquiring about it. The girl had trouble at first, but calling it some mix of white and blue. The point in that was that kids learn colours somehow related to other objects. And the sky, as "an object", is a very different category and was thus weird for her to assign a colour to.

Unrelated rant over

I, too, think about this all the time.

Maybe what I see as red is actually what I see as blue to someone else.

This is a very common interesting thought, but what I've started thinking is even more interesting is this related thought:

Why does red look like it does, to you? I'm not concerned with how other people see red here, I'm just thinking about a single person (me or yourself, for instance). Why does red look like that? Why not differently? Something inside your eyes or your brain must be deciding that.

You could say "oh it's because red is this and that wavelength" but what decides that exactly that wavelength looks like that (red)? There must be some physical process that at some point makes the qualia that is red - but how does it do that? The qualia that is red seems to be entirely arbitrary and decidedly not a physical thing. It is just a sensation, an experience, a qualia. But your eyes/brain somehow decides that ~650 nm wavelength translates to exactly that qualia. What decides that and how?

You can test this: https://ismy.blue/

Obviously not perfect since it depends on screens and lighting conditions.