I think a very simple solution is just to enable bandit infighting. Bandits attack one another and gradually build up troops by recruiting prisoners and leveling their own units.
When a band gets big enough they can form a larger party of brigands led by a Wanderer as a notorious outlaw and becomes a new Bandit Minor Faction.