I normally use what speed the miller is on, it doesn't seem to matter. But I would err towards the higher speeds.
The scuffing could be caused by a few things, very soft material, not enough lube, or just pressing down too hard, but I suspect there was a build up on the bottom of the cutter, of ali swarf. What you can do is put a bit of rough W&D with lube, between the cutter and the job every so often, and go thru the cutting motions, just to clean the bottom face. I normally check the quality after every line, then if it is a bit rough, clean off and go over again.
You don't have to use the cutters I gave you, there are lots of ways to get the same effect. It is just that I find these much better than everything else I have tried.
John