The size of a lord's party also depends on how lucky he was lately with rescued prisoners, as well as how much wealth he has. Check their info - those lords with very few troops probably have wealth in range of 5-10k aurums.
Which damage difficult should I play on? Normal to me and allies?
That's up to you. Nobody will judge. My personal recommendation is always normal damage to allies, because otherwise you won't really experience the full potential of army battles, and won't have to utilize your tactics/units to their fullest potential.
As for other settings? That's up to you.
I personally have everything set to max, except damage to myself (1/4 - 1/2 usually), and automatic blocking. (I suck at blocking, and most units in Perisno attack like a hurricane, which means I'll be lucky to block one swing out of ten)