(mild spoilers for both series ahead)I think on an even playing field, the Dunyain would almost certainly outclass the Vampires, although there are a lot of issues one could bring up here, since the whole idea has a number of built-in nuances making it weird to figure out (like just deciding on what actually counts as an "even playing field" is hard to imagine).
Nonetheless I've pondered this numerous times, given that both series rank very high among my all-time favorite/most influential pieces of media, period. It's just a bit tricky though, since in the series with arguably more scientific grounding (Firefall) we actually don't get to see all that much of the Vampires in-depth, and certainly nothing on the level of the full-on POV we get via Kellhus. On the other hand, while we have a lot more general information on how the Dunyain mind actually functions and so forth, they're also just plain more fantastical. I'm not saying the IDEA of a Dunyain-like being is pure fantasy, but in my opinion the abilities they have definitely cross further into the line of fantasy than the Vampires do -- although to some extent this might depend on if we're talking about any old Dunyain rather than Kellhus, who is unique even among his own kind (at the very least, a prodigy).
Ultimately though, since both "species" have degrees of realism mixed with speculative ideas (which both authors then sort of just run with according to what their plots demand, using the "smarter-than-a-human-can-comprehend-anyway" flavor of handwavium as needed), I think the Dunyain would probably curb stomp the Vampires. Whether or not this would be an immediate or easy victory is definitely up for debate though, and again brings us to the issue of the context of any such showdown.
The biggest weakness the Dunyain have (discounting anything spiritual because otherwise this whole idea just doesn't really work) is basically their individual, initial exposure to the world outside of Ishual. Beyond that, their only other notable susceptibility to an opponent is the opponent
actually realizing that they're a Dunyain. One of the main advantages the Dunyain have is their ability to appear completely human (and thus weak), whereas the Vampires seem to be almost the opposite; the humans in Firefall can instinctively tell the Vampires are "something not quite human" literally just by being in the same room with them...but then, in neither Blindsight nor Echopraxia were the Vampires actually attempting to be deceptive about their identity. Nor do we know if the Dunyain could pull this trick off on a Vampire anyway (considering that the Vampires can
literally smell cancer, I'd give them the advantage on this one based -- they might not know what the Dunyain
actually is, but I don't think they'd be fooled into assuming they're just another run-of-the-mill human).
The biggest points of possible contention mainly stem from the fact that, for the audience, the Vampires simply aren't as fleshed-out as the Dunyain, ranging from what they are/aren't capable of (be it in terms of superhuman intelligence or just stuff like strength, reflexes), to whether the Vampires are even sentient beings --and if so, whether that even matters.
From what we do know, however, there's really nothing I can recall off-hand that gives the Vampires any particular edge over the Dunyain. The Vampires do, however, have at least one major weakness -- the geometric pattern of a crucifix causing their brains to fuck up.
But then, again, this an issue of context, since the Vampires we actually see on page have a drug (I think it's a drug anyway, been a little while since I read either book) that negates this handicap. So if we're going to attempt putting the two on an even playing-field, I'd say you'd kinda have to include that in there, in the same way we'd have to assume the Dunyain in this scenario are already at least somewhat acclimated to the world-outside-of-Ishual.
However if we drop all pretense of an even playing-field, and just assume the Dunyain are all still in Ishual with little-to-no knowledge of the outside world, and that the Vampires are functioning without their anti-crucifixion-induced seizure drugs...then I'm pretty sure the mere geometry of Ishual itself would fuck up any Vampire attempting to invade.
Probably the most interesting match-up would be something like Kellhus, post-Leweth but pre-Atrithau, encountering Sarasti in the wilderness. It's the closest thing I can think of to making it "fair" for each side. As someone above mentioned, preparation is a big element here, so it's easier to assume neither Dunyain nor Vampire have any knowledge of the existence of the other species, nor what they are -- or aren't -- capable of.
ETA:
Blindsight spoilers
Wait but we never see a vampire in Blindsight.
All we see is the ship's conciousness manifested in the body of the vampire... right?
I'm still genuinely unsure how to parse out where one ends and the other begins with this issue (I feel like Watts has answered it in an AMA or an interview but I've no idea where). In any case, the Vampire we
do see in Echopraxia seems to operate essentially on the same level as Sarasti. If anything the Echopraxia character could be SMARTER than Sarasti, although this impression is likely more just because we don't get all that much in-universe exposure to the Vampires period (relative to the Dunyain, that is).