I've only noticed in the recent playthrough, but whenever the deserter party is spawned, it looks like it has no food, so by the time they get to the village, they lose a lot of people (more than a dozen from what I saw)
Not sure if that's intended behavior