Sounds like one of two things. Both are the same problem with different cause, mounting error. This is especially true if they hit dead center if you aim it like a rifle like you said. Good news is that only requires a modified technique , not a modification to the stock.
One, is your head down on the comb?
Second, I would imagine that your stock is a standard rising comb. If you place your head at varying distances from the tip of the comb, it will change your point of impact. I put a piece of cheek-eez where my cheekbone should rest on the stock. This helps me keep a consistent gun mount. Try mounting your gun with your cheek closer to the tip of the comb and see if this helps.
