Vintage RTO tools were designed for use on paper case hulls, and some of them make a better finish on paper than plastic.
Many do work satisfactorily on plastic hulls but better results are achieved with hulls which have a smooth and Round case mouth and edge. Your pictures indicate the pre-rolled case mouth to be rough and also with an outer flare and not truly round - this will tend to produce an irregular rto. With practice you acquire a "feel" for applying pressure to the shell as you crank the handle but this becomes difficult to sense if using poor condition case mouths. As pressure is applied during the roll procedure, the out of round segment of mouth edge often rolls outwards instead of inwards.
Also the finished case looks to have insufficient depth of turnover. I would go for 1/4 -9/32" free tube above the overshot card to achieve a goood, firm turnover.
Check the condition of the pins in the RTO head, also that there is no undue "wobble" of the rotating head (many of these old tools have had a lot of wear). Off-axial rotation will contribute to forming an irregular finish.
Hulls with a thin skived case mouth eg new Cheddites can make it more difficult to achieve a good and firm RTO finish most notably if there are slight variations in stack height of the load.
Invest in a good case trimmer or make one....see Creative Reloading Solutions website.
A smear of vaseline round the outside mouth wall of plastic cases or a rub with beeswax on paper case hulls will help you.
