You're certainly not the only one.
It's a lot like that here. They tend to wave back the least out of any other group that I encounter. Most times, I too get the dirty stare down. There are a small number of sailers that are okay, though. Once in a while, I get to talking to a sailboater and they seem genuinely intrigued in our boats and are nice people. Likewise, given the benefit of the doubt, I give them a certain respect for what the enjoy takes great skill and knowhow.
A lot of times, however sad, most do seem to turn their chin up and not bother acknowledging you.