The load has some bearing on this question, no? Are you shooting 1 1/8 oz through a 12 ga. or 7/8 oz through a 20 ga (or something else)?
Interestingly, none of RST pigeon loads (even the 3 3/4DE 1 1/4 oz load) come in 6s, but maybe that is because the distance to the edge of the ring is a closer shot than you are talking about.
Per
Randy Wakeman #9 shot launched at 1200 fps has only about 7/10th of a ft. lb. of energy left at 40 yards; #7-1/2 has 1.3 ft. lbs., #6 2.2 ft. lbs., and #5 3.1 ft. lbs.
That indicates that #6s have 70% more retained energy at 40 yards than #7 1/2s.