Engine now runs on a cheapo tea candle. I will probs my do some more tinkering with it, but I'm happy to get it running ad it was intended!
I ended up pressing the flywheel off and giving myself a little more gap between the spindle and flywheel/crank assembly. I figured that would give it some sideways play if needed. I then took the joints off of the upper arm and lapped them again with lapping compound. This time I made sure I lapped with the arm right side up, up side down and flipped both directions. That really seemed to smooth it out. I think I was getting some drag from the center pivot point because when I flipped the arm upside down in that slot it bound up.