There is a bi-annual flyer shoot held about an hour from me in SC at Broxton Bridge Plantation. About 6 years go the SHARK (Showing Animals Respect and Kindness) and LCA (Last Chance for Animals) people set up on the highway outside the plantation where the shoot is held. They flew a drone with camera over the pigeon rings to get camera footage. The owner told guests to shoot it down, that they had no right to be filming over his private property. They did, with rifles. Nothing ever became of it, and the animal rights activists never came back that I know of. You can do a search for it on youtube and watch the videos.
If someone flies one over my house/yard/hunting land, intentionally spying, without my consent, it's getting the same treatment. Sometimes it's easier to get forgiveness than permission.
FAA regs state:
"If a drone is flying over your property and taking pictures and videos of you without your consent, then it is violating the FAA’s privacy guidelines. However, you would need to be able to provide evidence that a drone is taking footage of you in order to prove that it has violated the FAA guidelines, and this is incredibly difficult to obtain.
There are some restrictions about where drones can fly, including stadiums and sporting events, airports, government-secured airspaces, and the greater Washington, D.C. area".