Ive never had this happen before because NORMALLY the more you trim the faster you go...However, I just put on a new prop today and I actually go the fastest with just a little bit of trim...
4500 rpm = 55 mph
4700 rpm = 54 mph
4800 rpm = 53 mph

My engine should be operating from 4600-5000 rpm so I guess Im wondering if I should continue to trim knowing I will loose speed but gain rpm?

That would mean you have the wrong prop for the boat. You trim for max speed.

Trim for speed - read this it explains it somewhat

http://www.go-fast.com/prop_slip.htm