I just got back from my maiden voyage with my new motor.
Let me run down the specs
26' checkmate new 540 -575 hp.

I went out and Powered up and the boat seems to be pulling to Port. When I throttle back the nose seems to dive and the boat rolls pretty hard to starboard. The boat seems a bit unstable.
Could this be due to the new hp?

Any insight would be appreciated.

The first questions I have is not about horsepower, because I know that hull can easily run that power. Is there any play in the gimbal? Do you have hydraulic steering?

Are you running a right hand rotation prop? Does the boat lean to the left under acceleration? If so it could be engine torque, lots of singles run left hand rotation props to counter-act the prop torque. Does dragging the port tab help?

I knew a guy that was having similar troubles. Boat was pulling all over on accel and deccel. He put it on the trailer and found the gimbal was cracked, around the tiller pin at the top. That is where the two bolts stick out of the gimbal. Take your drive and move it up and down and back and forth.. Should be No play in a perfect world.. That's what boating is "A perfect world"..

Hope this helps.

I have the same boat, and have ran it with more H/P than that and have no issues like that, so something is wrong, I do not have hyd steering and did have handling issues in the 70's and pulled the blower motor out...Rob

A few more facts:
I was low on Power steering oil so I topped that off seemed to help.
The gimble seems smooth and no play
Currently have a 4 blade 25 p prop on there to start.
Looks like starboard trim tab is stuck down port is stuck up. ( is this enough to cause the issue?)
The steering feels a bit jagged when turning left to right and back it feels like a worn out cable. I know its not a cable but that is how it feels.

Also I do not have one of those little sacrificial zinc fins like on most outdrives. Is it necessary? I am new so be nice!

Sounds like the trim tabs are the problem! They definately need to be working correctly for safe operation at high speed! Also sounds like air in power steering or loose belt.