Flint is grey photo degradation affect red purple blue first
Ever seen an agate fade? Petrified wood? I think there are lots of things on earth that don't fade. Lots of them are minerals and so are case colors.
Bakers have more "durable" colors because their owners would rather hunt with their Parkers and Elsies and Foxes instead. So the Bakers stay home and don't see the wear.
Acids and abrasion are the two things that "fade" case colors. I've never tested gun oils for pH, but I would wager than a few of them are on the acidic side slightly. If so, that would do the job over time, just like the oils and sweat on your hands may do the same thing. Grit is pretty obviously a factor since the high spots fade first.