rleete,
Over the years I have had my 7x lathe I have acquired 3-inch, 4-inch, and 5-inch chucks, so I can speak to this.
The 5-inch takes too much away from the motor, so ends up not having very much torque left over for cutting. It also makes a heck of a flywheel and keeps spinning when you...