Probably not. Keep your loading area clean and you should be OK. Don't allow build-ups of lead or powder dust - clean that loader frequently.

Bullet casting is the activity requiring the most care - both for lead fumes and for burns.