I would suggest that they are allowed the option to pull from castle garisons, or city recruit pools. These magic troops make the game both easier and harder, easier because I can get an unlimited supply of loot and EXP by just camping outside a castle, grindyer because their armies come back the next second.
I would sooner see a fix by reducing bandit spawn, or just flagging the lords as untargetable by the bandits? or as above, let them pull troops from the garrison, but the troops need to come from somewhere.
An another alternate snowbal fix may be letting cities revolt, but this is likely a different topic.
Or let them 'send for help' in some way but no mechanic for this is in place.