I am looking at several boats that need work, a couple of them need headliners. Has anyone ever used Gel Coat for a headliner? Is it possible to apply in this area?
Is it a terrible idea? Cost prohibitive...

Anything can be done if you wanna put the time and effort into it. If you saw a boat with a gel-coat headliner, i'd bet it was the outer skin of a part pulled from a mold and glued to the underside of the deck....Not the actual deck of the hull. I don't think any builder would put the time into spraying the gel and then sand all the orange peel out of it and buffing it smooth...way too much work.
it would be even more work to sand all the glue/foam off the undersides just to get to the point of being able to spray the gel on....then again sand all the orange peel out and then buff smooth.

Ughh, that sounds like a nasty dirty job. Call Mike Rowe, maybe he would take it on, except I don't think he does that show anymore. Why not just put a new headliner in it?