Hi,
I can cut off a 2" piece of aluminium on my little Taig lathe in about 5 minutes provided the the cut is as near to the spindle as possible. Aluminium loves to stick to the tip of the tool and make it very inefficient, I like the HSS T-bar that comes with the Taig's rear tool post, it really cuts well once ground and honed properly.
Try a decent and sharp HSS bit and don't let it extend any longer than necessary to make the cut or you will get chatter as the blade flexes under load, WD 40 works well with aluminium as a lubricant so does paraffine. If possible try and support the material by a live center in the tail stock to prevent the cut pressure pulling the stock out of the chuck's jaws, this is one of the reasons that you do not get a straight cut also lock the saddle in position to increase rigidity.
Good Luck.
A.G