1. they do not always leave the hideout
2. You can't always wait for them to leave, the quest attached to the hideouts only gives you what, 7 days? 10? Not nearly enough to wait them out.
3. It makes no logical sense that I would only take up to ten people with me when I have an army of hundreds.
4. I can't pick which guys go with me, so I'm often left with a bunch of fresh recruits backing me up on a difficult mission instead of my elites.
5. When you can't wait them out due to time constraints, the difficulty spikes exponentially, without the reward matching the new difficulty.
I could probably come up with more reasons if I stopped to think about it, but just this enough suggest that the hideouts are very much NOT fine.