Is your motor trying to walk out of the water. Mine would do that when it was trimmed to high. It also had a lot to do w the boat I had it on too. It was not a good match to the 1644 GD round Chine and coco did say I'd have trouble w it

As Sammy says, make sure it is not trimmed too high. Give us some info on the boat. What size and what kind of bottom? Does the boat have a tunnel? Is there a hook at the transom? Every time I've seen this it was caused by an unbalanced load from side to side and/or a boat that had some deadrise in the bottom. It would try to turn like a snowboard. Try taking a buddy along and have them move from side to side. Start by putting more weight on the left.

make sure motor is mounted on center of the transome. If off by an inch or two it will pull your arm out of socket.

almost looks like a short transome model hard to tell though. A short transome motor on a tall transome boat might would do this.....maybe? Never ran a setup like that.

Good looking rig. it ought to run straight as a string with a neutral handle. I'm assuming you aren't the first owner since it appears to be an older engine. Does the transom saver reinforcing plate drop below the bottom of the boat? Even 1/16 of an inch can be felt. Is that a Go-Devil prop? Since Go-Devil hasn't used a 2 blade for 3 or 4 years it may be worn or another manufacturers prop that is causing the problem. At what rpm does it start to pull? Good luck. That rig should run like a scalded dog with only one person in it.

After looking at your pictures again it appears the seat and battery are both on the right. Did you try putting more of the load left? Some boats are really sensitive to left/right balance. A grab bar would help when you stand to balance the load. A 20 X 40 boat I had would change the tiller forces by moving a gallon jug of water from one side to the other.

If weight is not centered right in the boat it will pull

A side or quartering wind will make it pull as well. I was told that you can bend your skeg a lot, like an inch or more, though you can also shift your motor if it's severe. Even a perfectly straight running machine will do this in a quartering wind or poorly balanced load though.

When boat is running in a straight line look back at your motor and see if it is running true with the boat, if the handle is real close to you...you need to bend the skeg a little more until the motor runs true with the boat given it is centered on the boat transom, which ever way your motor is pulling bend the skeg the opposite way