When we reach a certain population (prosperity) we should send out homesteaders , decreasing the prosperity/population significantly and spawning a new village that of course feeds new food to the town. Assuming you couldn't set up better trade routes or just make more local farms, that what you would have to do right? Entice the abundant population to move out to a new prospect.
Unless TW intends for prosperity to not represent population and rather it's just the same people eating more and more food, a whole town full of mukbang ladies, gobbling up the food and starving the garrison.