Just depends where you want to cast your doubts.
Since they apparently had minimal means to support a large population, until we get information otherwise I'd rather assume they don't have one.
2000 years of digging is a long time, but you'd need quite the army, slaves or superhuman worker-bees, if 100% of Ishual was built by the dunyain. 10 or 10,000 would likely not be enough.
For sranc, Koringhus survived for years on his own caring a child. That miracle aside, assume the others are nearly as competent. A very small army of dunyain would be able to hold off an indefinite number of sranc for a long time. Its not about killing them, its about living. And live they did. A large population, again, needs sustenance and is harder to hide. Smaller is more likely to succeed in this case than larger.
"Like he didn't know them" - or like they were not worth remembering since they were failures. I doubt they kept close count of the defectives.
Per giving them up - the opposite to what you said. They need to keep the population small. Once a superior specimen is available to replace the older one, the older one would be sent out or killed. Kellhus replaced Moenghus, Koringhuis replaced Kellhus.
"Many lost their lives" is a relative term. If you've got 10 people, 4 is many (not to suggest only 6 went into the halls). If you've got 10million people then 10,000 is negligible. No way to use that for numbers.
They would never use defectives as a workforce, waste of resources. We know that only the best, most agile of limb and mind worked on the halls - it was a form of meditation and training, not just some monument building exercise like the pyramids.
Multiplying my number by 10 is too arbitrary. Why stop at 10? How about 10,000? 10 million?
In addition, we've never seen flashbacks or memories of any large gatherings. Everything has been left to a small handful. If there were hundreds of people milling around, harvesting food, excavating the Halls, etc. etc., I'd think that there would have been at least some descriptions to that effect. Ishual is not a huge place, all those people must have been somewhere. [edit: obvious answer: they were all in the halls always]
Now, if we start with the assumption that they had the means, whatever they were, to support a relatively large population, then hundreds is not unreasonable. Ishual wasn't particularly well hidden, a large population would go undetected just as well as a small one as long as no one was looking. If they had the resources, larger groups should be able to fight off the attackers. If there were enough of them and they never ran out of food, then they should have eventually killed all of the attackers. Clearly that wasn't the case, so there must have been few enough for them to lose.
In my mind, it makes sense for there to be less than 100 rather than more. Big groups of Dunyain doesn't jive with my psyche for some reason. Get too many of them together and you've got an unstoppable force, especially with home field advantage in the darkness of the halls.