to have them in different tiers 1,2,3 and maybe 4
tier 1 being night operation
tier 2 rushing to destroy the place
tier 3 you would need to siege it as the bandits have fortifying the hideout which would be more difficult then normally
tier 4 can be that they siege engines as ballista or...