I found some new information. When I took a quest on village to destroy this hideout (taken that I will destroy it) it just stayed with 41 bandits. BUT when I took the same quest, but on my companion and his 11 men the hideout now only had 5 bandits. So I came up and destroyed it. My companion also completed the quest.