That could be the case, but I keep thinking that, even if descended from several Nonman/Man hybrid lines, those genes would be very diluted by this point. Then again, this is a fantasy world we're talking about, and it may not matter how many generations removed they are from one or more Nonman ancestors, that blood will always have an effect. (Couldn't Harapior sense the Nonman ancestry of Serwa, too? And that would - as far as we know - come only from Omindalea's son, who lived some 3000+ years prior...)
It definitely would explain why these women are apparently so rare in the general population, and why Kellhus could only find Esmenet despite being able to pick any woman he wanted as one of his concubines.

It might be genetic, though it's quite far-fetched, since women carry mitochondrial DNA (mtDNA), which is passed unchanged (save some random mutation now and then) from mother to daughter, never from the father. So it might be that some mtDNA strains are more compatible with Dunyain seed. It would be even better if we had any knowledge of a Nonman woman (^^) ever bearing a daughter conceived with a Man man.

I've always liked the idea that they first did something akin to the Womb Plague on themselves, a long time ago, and that when they travelled through the Void they were already Erratics. No real confirmation on this point though, and with the Tekne they may have circumvented the memory issue, at least to some extent.