First try some duct tape over the safety to see if it is you or the gun.

If you, you are on your own.

If gun, try a gunsmith who works on sxs. It is quite probable that even if you are touching the safety too much, the safety isn't performing as it should. Recoil is likely not the issue. Makinson has done a few of these and says there is probably a bit of metal slippage or wear that can be easily fixed. This seems to be more of a problem on subgauge guns for me so give it a try.