Did you do a leak down test on the motor? It sounds like you should be getting more from that set-up than you are.
The way that it is running now i would say that yes, you are over-propped. If there is no rev limiter to check I would think you want to be at 5100rpms.
There is alot of guessing here...you best bet would be to get the motor on a dyno and see what kind and where it makes its HP.