As the time goes there is too much bandits on the map. Since there is no headhunters to hunt for them and lords being imprisioned all the time. Make bandits fight each other. Looters>forest bandit>sea raiders etc. or even different "clans" fight within their own bandit faction.
