Yes, as a general rule. The case hardened layer is too hard to cut through with any ease. Standard HS and other graver material stock like Cobalt, etc will fail quickly on it.
Though using carbide gravers, you can cut through the case hardened finish quite easily in most instances. That glass hard, brittle top couple of .000 'case' breaks up and chips instead of smoothly
cutting away. That can leave a less than fine finished look.

Also, some guns that look case colored actually have no case hardening to them.
These can be easily recut w/o annealing. Just polish and recut.
Check for the 'case' with a sharp file in several locations on the parts.

Most case color providers will advise to anneal before redoing the CCH anyway.