We can tell what’s green and what’s non-green so I’d say we both “actually see the color green”. We might see green differently (e.g. if our brains switched maybe how you see green in my brain is how I see cyan in my brain) but we’re both 100% still seeing green.

What I’d be more concerned about though is sound, because if all the colors are switched around that’s okay with me, I can just figure out which color corresponds to what, but I’m not sure if I could do that with sound.