I am redoing my engine compartment I want it butt smooth and shiny, would bondo or high paint buildup be best ,and what would be the best products to use

I always use gelcoat ( topcoat /gelcoat with parafine) so it don,t stay sticky like gelcoat does if not covered.
its not high gloss ,but its a strong and chemically a perfect coating that can resist gas /diesel/oil etc .
You can also use a urethane coating ,these are chemically strong and can resist gasoline/oil etc ,these urethanes are high gloss if you want .
Sand it with grit P 80 if you use gelcoat /topcoat, and use P 150 for urethane ,preferably with a exentric sanding machine .
Old holes can be filled with epoxy ,I use empty sealercans that can be bought ,you can put the epoxy in ,and fill those holes that are not used anymore.
Don,t forget to grind all old sealer out of the old holes so the epoxy can get a good bond .