A soft oily cloth should move most of it.
Fine wire wool, some say to use it dry as it's been said the rust and oil mixed is supposed to be more abrasive than the wire wool. I'm not convinced though and have used it carefully with oil and had no problems. If there's no stubborn bits after a good buff with an oily cloth it shouldn't be an issue.

Rust never sleeps !