I have no idea what specifically created the super farmer party you refer to. I do know, however, that if you intervene on a fight, and hire no helpers from the defeated bandits' prisoners, they combine into one stack. Hypothetically, for example: You come across a sea raider group with 8 farmer prisoners attacking a (X) sized farmer group. But... you don't care to hire the freed prisoners once you stomp the sea raiders, so you close the prisoner screen after the battle. The farmer group will absorb all the prisoners you freed but didn't employ. I'd assume the same is true for AI lords (who actually DO intervene pretty regularly from what I have stopped to observe, if not busy doing something else). I've made super stacks of 40ish farmers (usually only farmers of course) this way myself.
So... what probably happened is a hugely hugely rare hire-spurn-so-make-super-farmer-stack type of occurrence that started much like the above. That's what I would bet on, anyway.
Although... now that I think about it (some beers blocking teh intellect!)... you are talking about prisoners, not a farmer group traveling with 30 vaegirs actually as a member of the traveling farmer group. I'd assume it is nevertheless rather similar (and similarly very rare!), but still.... probably just a very odd result of AI behavior/farmer help.