If you refi you only pay interest on your current balance, only hopefully at a lower rate.
$100,000 boat loan at 6% you accure $16.44 in interest per day at that balance. So if you have a 240 month loan and a payment of $712 a month, that first month $493 goes to pay interest ont he $100k and $219 goes to principal.
When you pay your loan down to say $50,000 now interest only is accruning at $8.22 a day and when you make your payment a lot more is going to principal out of it than to interest. or out of that same $712 , $246 goes to pay a months interest and $465 goes to principal.
Now if you refi that $50,000 at 5% only $208 goes to interest, so a refi is saving you money. Assuming you doen't refi that $50K for another 20 years but keep the same term as what remained on your old loan