Kook - do not randomly remove your supply/return ductwork. 1st, assuming you have a squirrel-cage fan, check the full load amps (FLA) draw against the motor plate data. You should be 85-95% of that rating. Before you go any further, you should know were your at on the units fan curve. You have to maintain the velocity in the ducts to supply your registers. Velocity is determined by the volume of air and the cross-sectional area (duct size). It sounds like you have pre-insulated duct, which is fine. Are your 6" take-off's on 90 degree angles with the main trunk?, this adds considerable static pressure and can cause airflow inbalance, pretty common in residential.
The above amp reading advise assumes your unit is properly charged and your coil is in good shape.
ps. typical design for residential A/C is 1 T(12,000BTU) for every 400 FT2. Feel free to e-mail me after you take some readings if I can help.