I get decimated as well. However, When I was trying to figure out how I was going to take on 30 sea raiders with 8 guys i discovered if you kill as many surrounding parties as you ca, it seems to cause the garrisoned troops in the hideout to start leaving? So If you kill any forest bandits around I think that'll help.