one the great ME authors of yesteryear, don't immediately recall which one, successfully argued (to me anyway) that the need for these super high speeds for the very tiny drills is hogwash. I cant' remember the exact argument, something along the lines of as the speed goes up the downfeed would also have to go up - both trending toward infinity as the drill size gets smaller and smaller. obviously at some point a practical limit is reached - if we did have a ceramic bearing spindle going at 70k, you might that it didn't do much more for you than something at say 20k or even 10
The author i believe argued that there was a diminishing return as you got above a few thousand rpm
Now, I use a dumore speeder which is going at 25k, so I'm not adverse to speed. The point of the above isn't that speed won't work, but that you don't' need speed to do a good job. The critical thing is that you don't exceed a very tiny chip load - speed is sort of built in protection in that regard. The other way to get there is sensitive equipment where you can feel the drill.
here's a hand little device for achieving this - its a counter balance drill press table. you tape or otherwise fix the work to the table and the table raises the work into the drill. to prove the point i drilled a very small hole (90 iirc) using a 18" buffalo drill - that sounded boastful - wasn't meant that way, only that if the set up is sensitive enough, you can easily drill small holes at, relatively speaking, slow speeds (albeit it wasn't .125 deep!)
here's the table, its from a HSM article years ago