FWIW, I would polish to at least 400 grit if not 600-800. Sort of depends on what you want and how you will be doing your rusting. If you go slow with a dilute solution and not too much time in the humidity box (or no humidity box at all), you can keep a much nicer (in my opinion) luster. 320 is far to course for my tastes.
You are polishing with a hard backer of course! And from muzzle to breech and back again...
Also, you can touch the barrel all you want while you are polishing it. I use WD40 as a polishing lube, so everything gets greasy. Just wash in the hottest water you can stand, with good dish soap and rinse well. Then it is "no hands"...
Brent