I am sure this is a common problem, it will only trim down if the motor is in gear and helping push it. Also seems to be getting worse as time goes on. Do I need to change the oil in the pump, is it a trim cylinder seal problem?

You may need to rebuild your trim cylinders. I have done it on several of them. It gets corroded where the shaft slides into the end of the trim ram, mercruiser sells a rebuild kit that contains the parts needed it's around $35.00