Mark, at least in theory, annealing is a reset to "original" properties of hardness for the alloy used.
DDA
The hardness on annealing may have a similarity to the original part, but shouldn't the case, by definition, be a different steel than the original?
If the original alloy for the part is low carbon steel (likely), then there is no hardening from heat treat and no softening from annealing. The case has higher carbon content than low as it does harden. If it will harden, then it will anneal (excepting some exotic alloys). So, if we anneal a part, the case is made softer and the core is unchanged. It is possible/probable that the case will get higher in carbon and deeper with repeat cycles. However, if the anneal is done properly there should be no reason for the case to become brittle.
Maybe, the original question had to do with each case color cycle adding some bit more carbon and increasing the resulting depth of the case?
Likely.
Are we thinking that annealing removes the carbon that created the original or a subsequent case?
No.