Thomas, Zarra will have to assume that if a combat is about to start to don the glasses. Fay will always draw his sword for battle and Oswald uses a lot of spells that can really light things up. In the heat of a situation like this other characters will just act many times without giving a warning. They will be focused on this golem for example.
The old golems were immune to spells that allowed for spell resistance, the 5E versions just have advantage on saving throws vs spells.
So not easy to hit with magic, but not impossible. The only thing they're flat out immune to is nonmagical physical damage.