There is one way to handle this, and only one way: access by permission only. It's common courtesy. The "few" will always be with us. They'll turn your land into a silvicultural slum within six months. It works for me. I don't take crap off any one.
(Litle feisty today, returning from three weeks overseas.)

A young friend a few years ago told me of meeting a couple deer hunters on a property near mine. My friend said he had no luck and was going to go over to King Brown's. "King Brown?" they said. " You go on that effing land and that effing guy will call the effing Mounties in five effing minutes."

I have no gates. I don't fence to keep people or animals in or out. I expect those who want to use my property for any reason to treat the land and me with respect. The local bishop, a friend, trespassed on a promontory of my land 20 years ago. I told him there were 10 pairs of beady eyes watching our confrontation from the beach and that he'd have to go back for permission at my house.

He did, and I did. Stewardship demands no less. Otherwise, we're giving in to the galoots, to suffer the tragedy of the commons.