The most critical change I want to see is for towns and villages to lose a portion of a tracked amount of population whenever their volunteers, mercenaries, or conscripts join an army. In other words, if you or the AI keep recruiting troops over and over, it should seriously impact the prosperity and long-term recruitment potential of the town or village.
If, for example, a village has 600 people, and your recruitment there is at 1%, without bonuses to increase it, 6 peasants will offer to join you. The village will then be down to 594 people, and at an optimistic/exaggerated/(fun>realism) 1% growth rate per month, it would recover to 600 in one month. If you or a faction lord have enough bonuses to recruit 5%, then 30 peasants will join and the population would drop to 570 instead, taking more than 5 months to recover back up to 600 peasants (replacing 5.7 per month: 5 new people, plus the 0.7 remainder being added to the following month). On top of that, there should be a "cool down" mechanism, so if a village has JUST been recruited from, it can't hand over more for a few days at least. Perhaps after recruitment, the maximum number of available recruits from the village should be capped at +1 per day since the last recruitment, so if you go back 2 days later, you won't get 6 more troops, only 2. That allows around 30 men per month per village in the long run, which would be 5x the rate of recovery for a 600 population village. Armies SHOULD be able to run out of available recruits to fully replace losses, particularly in extended wars or several wars in succession.
In peaceful conditions, the villages should recover soon enough, and then begin to grow and prosper; in prolonged wars with forced recruitment by one lord after another, they should begin to suffer and decline, leading to less troops being available for future recruitment efforts. Bandits should also cause a loss in population to occupied villages, although some of those peasants might migrate to the towns instead of being killed, abducted, or forcibly recruited by the bandits. Obviously, the numbers would need to be tweaked after play testing, to insure that the villages don't get depleted too quickly, or that they don't expand too rapidly and wind up with massive overpopulation in a few years.
There should be a number of modifiers for recruitment, such as:
At War - would give a significant positive modifier, allowing you to take emergency conscription measures in times of crisis.
Owner - would give a significant positive modifier for the current holder of the fief, and possibly for his liege lord as well.
Enemy - would give a significant negative modifier, as the local population tries to flee, hide, or otherwise avoid being forcibly conscripted into a rival faction's army.
Relations - would give positive or negative modifiers based on how much the population likes or dislikes you.
The final result would then be multiplied by the current population, indicating how many peasants are available for recruitment, and subtracted from the population if hired.
Running your own fief well, by protecting it and not over-taxing or over-recruiting from your own land (where you would usually have the highest recruitment bonuses), you should gradually see an increase in population and prosperity. Abuse it, and it should suffer. Depleting the opposing factions' villages should be a viable strategic option, rather than being irrelevant as in the previous games. I found it especially ludicrous when a rival faction was down to its last castle and looted village, yet its 6-8 lords could all recruit hundreds of men each from the same village, over and over again without completely emptying that lone village of its male population. It's difficult to take a last castle when all of its lords are gathered there with sizable armies, in addition to the normal garrison, and all of that can automatically regenerate in just a couple of in-game days. A depleted faction that's down to 25% of its army strength, and can't replace losses quickly from its meager remaining holdings, would be forced to either come out and try to recruit from neutral factions without being chased down and crushed, or else be gradually wiped out by repeated assaults on the castle where it's hiding sapping its remaining strength.