The downside to flirting with too-low pressures is poor combustion, i.e., unburned powder and marginal performance.
I once believed powder residue was of no concern as long as I was happy with the ballistics. Then I encountered two instances with the same SxS where unburned powder was dumped into the action when I dropped the barrels and extracted the empties. After about 2,000 rds I experienced failure to set of the triggers and, even worse, hair-set triggers causing the gun to fire on closing. Disassembly revealed powder residue, both burned and unburned under the triggers.
It's not hard to avoid dirty ammo, so I do. Pressure is my friend...and I like 8,000 psi as a minimum.