I have have a 540 bbc in a cat with an Arneson surface drive that uses a BAM transmission. The starter damaged the teeth on the flywheel and i want to replace it. I have never done deep work on a transmission boat. can i unbolt the trans from the bell housing and removed the bell housing in the boat? or will i have to remove the engine?

I guess anything is possible if it has front and rear offshore mounts, but Id pull the engine and do the work,

That seems that it would be a lot more work then just removing the engine and flywheel. (pull drive, transmission, bell housing, flywheel) I just pull the engine and brace the transmission with a 4x4 once engine is out.
No need to replace flywheel, just knock off the ring gear and replace it.

gotcha i have just never messed with a transmission boat until now

If you decide to replace the ring gear yourself make careful note of its location on the flywheel. Some are towards the back, some towards the front. Depending on the starter location. Most have a step that the ring gear sits on so its easy to figure out. When you go to put the new ring gear on put it in the oven at 450 degrees for about a half hour and it will fall right on. Its almost impossible to "beat" them back on without damage. Your machine shop should be able to get you a ring gear. They are around $20