I'm going to do some more testing tomorrow! Unfortunately I cannot test my Xperia X because the camera no longer works, but I can test a Nexus 5 and my original Jolla 1.
IR affecting the focus was my guess. Although the view in the camera doesn't seem to do the focusing back and forth that it does when trying to focus in the dark.
I know that dedicated cameras are not affected; I can take pictures of it fine with a GoPro, a GoPro knockoff, and a Casio Exilim.