I always pictured everyone without beards for most of the series, except the skin-eaters. I cannot site any specific reference though, so I'm probably just wrong.
As for the Scylvendi, I suppose it would either be clean shaven or none at all, not somewhere in between. I can't see the people of war caring much either way, but with their obsession with manlyness and their proximity to the North (i.e cold) I'd guess that its more likely that they have beards.