Mike,
I was taking off a few thousands of an inch with my drill press "mill". No, it's not made for that but for cleaning a nearly finished chisel mortise, it was okay, to a point, and I don't plan on doing it again, but there will come a time... and if I had a mill, I'd probably invent things to do with it. I might drill and tap for a scope or cut a dovetail or two, but I'm not going to do anything really significant.
I almost bought a lathe once. I believe it may have been an Atlas. That was a long time ago. Then, I realized that 1. I won't be barreling that many rifles, and that's what I would really want it for. 2. If I did get a lathe, it would have to be big enough to handle at least 36" x 1.25 or larger barrels. I don't have the space, the need, the skills, or the money for that.
Unfortunately, all the machinists I know live far away. I might be more interested in metal working, if I had someone near by that I could lean on for help and instruction. But then again, I'm really not going to get into many more gunsmithing projects that would require that sort of gear.