I was running a 454 built to 420 specs boat ran 68 gps with a 24 pitch bravo 1
Now I have a stroker 496 with a larger custom cam 552in/555ex; 11/32 stems ported and flowed; 850 sea demon; DUI ign; rpm-air-gap with 1" spacer headers and no mufflers; the boat has a ton of more power low and midrange it is much much quicker but top speed is the same. Shouldn't I be able to turn the top end rpm's more, and turn a larger prop?

Yes, it should. Something is limiting RPM. What ignition is it specifically?

It is the DUI marine ign calibrated for my engine

That tells us nothing. Are these mercruiser 496's? When you say "stroker 496's" that infers 454's 1/4" stroked to 496. The Merc 486 is a different engine altogether.

If this is mercruiser stuff, you're probably hitting the rev limiter.

The motor is a 454 stroked to a 496 with an eagle balanced rotating assembly The rev limiter is set at 6500rpm I am not even close to it. It runs 24 degrees advance at 3000 rpm for a total advance of 36 degrees. The engine will not turn any higher on top then it did with the 420. Low and midrange is incredible. The only "mercruiser stuff" on the motor is the starter and alt. the block is chevy, everything else is aftermarket

I would think you'd want your total timing in by 2500 rpm. That has always been the general rule. So, you probably want to get your base timing set at around 10 - 12 and total between 32 - 36 by 2500 rpm. That could have an impact on your rpm. Did you dyno the engine? Do you know where your peak horsepower is?

No dyno all I told them I didn't want to spin the motor over 52-5300 it doesn't but I would think I would be able to over rev it some by using the same prop

The cam is almost the same as stock 420 cam. You could try retarding the cam timing to move the torque higher in the rpm range. I don't think the timing is affecting the performance

possibly the cam is in wrong. Does it have a multiple-position keyway?

What heads do you have on it? did you touch the heads? your original power bottle neck could have been the heads. Your cam is not much bigger than the original 420 cam, and adding ignition, carb and intake won't always get you all that much power. but the stroker kit will get you the increased torque you are seeing in the midrange.
JMO