abandon

I'm sure it's something with my perks or whatever but for the life of me I can't figure out why my max party size limit keeps fluctuating between 456-461. I'll recruit the max to 461 just to be told later I've lost troops due to abandonment. I'll check and my limit will randomly drop back down...