Can we just have a total of max looter party 100 from 5 - 20 looters(hostile to villagers), for beginning early game and for when you escape from being a prisoner to rebuild your army (farm). 20 max party from 20 - 40 looters, for when your army gets to big but still to farm (hostile to caravan and other lords). 10 max party of 60 - 80 looters for challenge to lord patrolling the lands (not hostile to caravan). 15 max party for special bandits such as sea raiders, forest bandits, steep bandits, tundra, etc. 8 - 20 bandits (their already well equipped).
And it will be like that for the rest of your game. By the time your super high level and have a huge army all the looters and bandit would just run away from you like flies.