It would appear that the gears were registered and transmitted torque through the mounting splines.
The screws just hold the two halfs on center, but don't drive the gears, so the load on the screws is very small.
I would wonder if you couldn't "model" that as a machined square journal, which will transmit quite a bit of torque without a lot of radial force. A hex would load up the screws badly in the radial direction.
Modified form spline teeth made with a small ball nose end mill would work, and not raise such a stress riser as a square corner, or square bottom splines.
So, a modified spline could be made on the shaft. Now what to do with the gear?
Well, you could split it, re attach, bore and turn the OD. Locate in an indexing attachment and cut internal splines with an endmill to match the external splines in the shaft
See crap o cad sketch
I don't mind saying that I would NOT look forward to making it..... :
Dave