The number of troops you can lead is affected by the tier of the troops in your party.
If your party size is listed as 51, it means 51 lower level troops. I think it's about third "upgrade" that troops become more demanding of your leadership.
I think the highest tier troops count as somewhere between 1.5 & 2 troops ... maybe.

I can't remember offhand.
It makes the Hidden Camp pretty handy, but you need to be careful on taking on the evil troops after the invasion happens.

Often times I'll dump everybody in the camp and tweak my army for best "bang for buck".

Fortunately, you get several days to fix the problem.