I'd take it a step further: Recruiting from the notable pool should only be possible for the lord who owns the bound caste / town or the faction leader which the lord belongs to. These are his/her levies. Nothing makes me more mad than seeing random lords empty all MY villages of recruits.
Hostile / neutral lords still have the option of forcing villages to provide troops, which incurs a relationship penalty. Maybe add a new notable for "adventurer" troops who are volunteers and can be recruited by anyone, including the player as a neutral option. They should upgrade into merc soldiers.
So we'd mainly see faction specific troops + mercs in every army, which is far more immersive, until settlements start changing hands and notables provide troops for their new overlord.