Blue Hylomar used by Rolls Royce and the CEGB on power-station equipment was always applied ton surfaces that were properly flat and finished to quite fine tolerances. It won't CURE ALL bad surfaces in any way. Also, all those machines and engines were "properly" bolted and torqued, where an adequate set of bolts were provided.
For all our clever ideas and alternatives that work on other engines, the immediate problem is advice on what we can see with these parts that don't work successfully. The comments about the line/groove at the "joint" between cylinder and side block seem most pertinent. The whole NEEDS to be "groove-less" - or filled with a suitable material (for temperature, expansion, gas pressure, joint pressure, etc.).
Putting it simply. The pressure (average) between the joint faces (applied forces from bolts divided by contact area) must be GREATER than the max pressure of combustion gases when the ignite at max compression. Suppose you have compression at 6:1 (measured). Then you heat the gas in the confined combustion chamber to (say) 800deg, C. You start with 6 bar pressure, then by "PV=RT" (Gas laws) you get maybe 24 bar pressure in the cylinder. Any "soft" gasket goo will extrude at that sort of pressure, given time. Suppose we have a joint face 1/8" wide by 1.5in long: I~.E. area 0.1675sq.in. Then we need a force of over 4.5lbs from the 2 bolts to withhold the gases at the joint. So I suggest you work-out what the bolts are doing on your joint area... They should be adequate, if to a proven design.
Personally, I think a setting gasket material may resolve the issue, but head gaskets are normally in a "hot zone" so destroy gasket goo by cooking it and breaking it down into rubbish. Paper (cellulose fibres from trees) can withstand the temperature better, but need no deep scratches or grooves in the surfaces. try applying the thinnest goo on the cylinder side, paper gasket on top and then fix the head on, The goo should only fill the scratches in the surface, the paper gasket seal over that, and hold the viscous goo from extruding.
K2