I think, if I understand your words, that you are making a mistake. you want to hold the block (the thing you are drilling) in a 4-jaw chuck wiht the place where you want the hole centered on the axis of rotation, and hold the drill in the tailstock. That way if the drill wants to wander, the work is rotating so it wanders in a circle which is a close approximation of the hole you want, rather than making a left turn and heading out to the edge of the block. Once you have the starter hole, ream it to exact size and then use that hole as your reference for all other work.