Unless you do something like rescue the village from bandits, they won't offer recruits unless you're a member of their faction. That limits you to companions and mercenaries until you join a faction, and to occasional freed prisoners from bandit parties you defeat. The start of the campaign tends to be rather brutal, until you can recruit and train up a sizable party.
Even then, it matters which faction you join, as the Russian and Lithuanian troops simply outmatch their equivalents in the other factions one-on-one. The Lithuanians aren't a playable faction and the Russians have a significant numerical advantage on top of their stat edge. I found my party could cut through a unit of Polish troops like a proverbial hot knife through butter, but got crushed by a Russian party barely over half as large as the Polish one.