If the hole is round you should be able to center things up to the point that you get no dial deflection. If not you are either off center or you don't have a round hole.
If I rotate the indicator just slightly the needle moves. I generally "eyeball" the indicator in both X and Y directions but is this good enough?
Bad dial indicator maybe? The thing is I normally expect a bit of a jump in indicator position when I first start to tram but that should be it. Afterwards you should see no significant movement as the tip of the gage travels around the hole. Do make sure though that you are actually in contact.
I had to add a C'sink to an existing hole but when I center the part using the indicator the C'sink was offset to one side.
Am I doing something stupid?
Stupid no! You could have made a mistake or your gage could have a problem. One gotcha is running a tram with the tip not actually in contact with anything. It is something I worry about as my eyesight has gotten worse. A funky gage can be an issue too, always verify that the gage is working properly.
Oh don't do something like I've done which is to bump an axis handle and not realize it. That might be stupid.
One last gotcha is having the tip of the gage bottomed out so it isn't indicating anything real.
As a general note when you are running a tip of a dial indicator in a bore like this the arrangement is less than optimal. You as the machines might not be able to get a good visual on what is happening at the tip and you may have to move around significantly to see the gage. It is much more frustrating than trading a straight edge.
When in doubt you can check alignment buy putting a dowel in the collet and making sure it enters the hole freely before going on with machining. If something is critical doing some sort of verification of the setup is in order. Sometimes that verification can involve simply moving the indicator to a different point in the bore and rereading the gage.
Outside of a bore I might slip a feeler gave under the tip to verify that the tip is actually under pressure. In a bore this is hard to do so your next choice is to bump the axis (turn the lead screw a few thousands) in each quadrant to make sure the needle moves as expected. Of course then you have to move everything back to center. The idea is to verify that the tip is actually in contact all the way around and that the tip isn't bottomed out in some manner.
Your approach might vary from setup to setup. Maybe you don't want to move one axis for some reason. Whatever, sometimes you need to improvise.