I want to give an old trick , it was the first trick I learn on my first job as a paid for work.
I had my mechanic technical school degree , at my 21 years old in 1970, it should be at 18 but I lost 3 years.
Well , following the history , my shop teachers where former employee at the plant where I where working, I have to re drill a big press die base to suit new parts as the model was changed , I nee to drill a 2" diameter , as teach I did a small , maybe 0,5" , then 1" and after it the 2" drill, just when I star it begin to whirl , and a old man , former work mate of my teacher, say in loud voice.
"HEY, DID NOT YOUR TEACHER TO USE A FOLDED FABRIC or CLOTH , BETWEEN THE DRILL AN THE PIECE" , no, I said and it show me to put at least about a 1" thick of piece of fabric layers or cloth between the drill and the piece, and magic , it make a perfect drill like it was reamed.
For smaller diameters , 4 layers is enough, emery cloth can be used too , old leather from gloves also.
The fact is to fill the flutes.
I use it since that time , when I want to have a "perfect" hole.
Of course it need more power to drill, and is the best choose to make a hole on a thin piece of metal.