Fantasy Fest is a once in a lifetime but personally I would go to the Race and Columbus Day Regatta instead of Fantasy fest. Fantasy Fest is extremely packed so if you do not like crowds forget it. You can never get a taxi, hotels are packed and filthy. Race week is more casual and more fun. Fantasy Fest is like Mardi Gras. Now Columbus Day Regatta in Elliott Key is the event not to miss if you are a boater.
I have done them both, Fantasy Fest I will never do again and Columbus Day regatta and the Race I always look forward to.
The drive to the Keys is beautiful so the flight in to Miami is always an option. If it is affordable to drive in to Key West and avoid the whole Miami Airport and rent a car thing than do that. I would fly in to Key West, take a taxi to the closest scooter rental place and have the extra half a day to vacation. You will not need the rental car once you get to Key West.