You may have to end up buying new collets if you damage them, so a few stub drill may be more advisable. As to how big a drill is too big for a collet will depend a lot on the material that you are drilling and your methods. Step drilling, working your way up to final size using smaller drills first, will ease the load on the collet.
A lot of things that we do, although not advisable, we can get away with, so it's probably worth a try. I often see pictures of people milling with the cutter held in a drill chuck, I cringe every time that I see that, but they get away with it, it's only 3 point contact on the shank and certainly not the way to do it.
Paul.