For what it's worth, I think the food production to prosperity ratio is mostly fine - in that the limits to food production is designed limit prosperity. Arguably, the lack of buffer space between food the city needs to consume is a problem (such that, arguably, cities are too sensitive to civilian starvation / prosperity loss based on even pretty brief food disruptions), but, conversely, some would argue that the small buffer space makes the game more interesting and varied (e.g. raids are more consequential, so players are more likely to respect them).Fundamentally, though, we should expect there to be an equilibrium between prosperity and food production, such that the city eventually grows big enough that it consumes just about all of the food that it can. (If villages produced more food, prosperity would just grow to match, which would cause the city to consume more food, etc.)
The problem (as we're talking about above) seems to be when prosperity dips really low, such that food prices are really low (prosperity drives all commodity prices), such that merchants come and buy the food away instead of sell food to the low prosperity city, even if it's starving. There should be really strong in-elastic demand for food basically: The city should only sell it's last 100 food or so at really really high prices.
IMHO, then, the problem is not really the prosperity to food relationship. The problem is that the garrison gets its food, unmediated, straight from the same granary that the city uses for its civilian prosperity-based consumption.
I think there should be a separate buffer food stash for the garrison. This food stash would steadily buy food from the local market (for cities - would need to figure out an alternative mechanism for castles) until it reached a cap (e.g. 200), so it would still be dependent on the local food supply and ultimately be negatively impacted by general city / castle food shortages. When the city's food supply hit zero, such that the market hit 0 food stocks, there would be no more food to buy, and the garrison's consumption would draw down it's stash. But if the stash cap is say 200, and the garrison's food consumption is 10-30 or so, there would be more of a buffer to protect the players investment in the garrison. (You could play with those numbers of course in testing if that turns out to be too much garrison protection.)
