The problem is the way looters/bandits spawn in M&B, Warband, and probably Bannerlord. There is always a fixed number of looter or bandit parties of a type, so when you destroy one unit, another is created automatically to replace it. That means, no matter how hard you work at it, you can never reduce their numbers, and no matter how much you ignore them they will never expand into a real threat.
The solution, in my opinion, is to have looters (and some other bandit types) have a variable respawn rate, and a minimum time until the next one can spawn. If they can only spawn one unit every x number of days, then you can keep their numbers reasonably in check with an occasional patrol or a scouting party. If they exceed some number, the spawn rate drops to half or a quarter, and reduced substantially further above some soft cap, so they MUST be cleared every so often to prevent them from accumulating over time, but it will take a long time for them to become a major problem. If you do a thorough sweep, they will only reappear gradually over time, rather than just popping right back up the moment you pass.
I agree that looters serve a legitimate purpose in the game, but the need reduces itself over time to some degree. As factions become more wealthy and powerful, they should be able to release patrols, making looters relatively rare in the more prosperous areas. Having 4-5 large groups of Steppe or Forest Bandits all converge on a Lord from various directions should be an extremely rare occurrence, rather than happening as frequently as it does in Warband. I've jumped in to save lords from the combined armies of over 150 bandits on a couple of occasions, and seen bandit parties with upwards of 25 captive faction troops, in addition to captured peasants.
The present system makes eliminating looters/bandits pointless (aside from training up new recruits and the pitiful loot when you first start the game with practically nothing), since they simply reappear elsewhere.