Hi Terry,
On mine, went as large as possible without restricting the movement in the direction of the mill column. Sometimes you need extra room to hold down odd shape parts. Another consideration would be if the heights were different, and you would lose too much vertical travel. Amazing how quick a rotary table, mill vise, and workpiece add up to a lot of height.