Brownells sells a little grinding wheel that fits in the drill or drill press. They are extremely easy to use at slow speeds and will do the job you want.

Lenard