I think it would make sense to have the banditry in the area (chance of hideouts and bandit spawns) be related to the prosperity of nearby settlements, the more prosperity means less people in need, and less crime, it doesn't really make sense to see so many looters spawning near a town with 10k+ prosperity where so many job opportunities exist.
Also when a village is raided about 30% of the villagers should escape to nearby villages increasing their hearth size and towns/castles becoming militia, the villager groups would be visible in the map and can be intercepted if you are feeling specially sadistic. A few days after the village was raided a big group of looters representing about 30% of the initial population (or smaller groups of 5~10% each) would pop up and start being annoyances near the village ruins. (we assume the last 40% died in the raid)
Also when a village is raided about 30% of the villagers should escape to nearby villages increasing their hearth size and towns/castles becoming militia, the villager groups would be visible in the map and can be intercepted if you are feeling specially sadistic. A few days after the village was raided a big group of looters representing about 30% of the initial population (or smaller groups of 5~10% each) would pop up and start being annoyances near the village ruins. (we assume the last 40% died in the raid)