Rigidity of automorphic Galois representations over CM fields