It's not really a downward spiral, but the system is underdeveloped.
A big issue is people misunderstanding how the bonus from food in the market impacts things. It's quite confusing and I don't blame people for not getting it. The market bonus from food is part of the general town consumption model, where the town decides to sell off excess goods when there's low demand, it really has very little to do with actual food need in the town and gets messed up by other lords and caravans buying up the food when the demand and the price is low.
The main way that food is provided to the settlement comes basically directly from the villages. It's not the mobile villagers as was mentioned, that just delivers goods to the market, and only gives a kind of basic bonus based on demand. But rather a function of the number of hearths in the villages that aren't pillaged.
This won't be enough to maintain high prosperity. So those food related bonuses are required. A normal settlement with 3 villages at mid-ranged hearths will get about 34 food, which will get you to about 1700 prosperity without a garrison, with 500+ hearths it will go up to 46 food, or leveling at about 2300 prosperity. So you do rely on the food consumption calculations. But those are just a bit funny. Basically they're tied to the demand of the good, which is generally tied to the supply of the good, but so is the price, and as soon as the price gets low enough, caravans will come in and buy up all the stock to sell to another town.
Now, the thing is, demand doesn't go up when your settlement is starving, and if it DID, it would actually hurt the calculation because high demand for food means that you will sell less of it on the market so you'll get a lower bonus. So it's this weird situation where food in the market is tied to your food for the granary but only when it's cheap and there's enough of it, but if it's cheap, traders might buy it. If you drop it on the settlement some of it might make its way to the granary, but your prosperity also increases demand for goods so it still might not be enough to get as much bonus as you'd like.
So basically, your base static food is going to provide for about 1700-2300 prosperity if you keep your villages healthy. Everything else comes from this weird settlement consumption system, which is pretty counterintuitive, and while you CAN assist by delivering goods yourself, it's a bit weird. You can end up with a much higher bonus from having a reasonable amount of grain where the price has become 7 versus an absolute ton of grain if it somehow manages to be 14 because the whole supply/demand system can be a bit confusing.
Now, having 3 villages pillaged will essentially wipe out about 1200 prosperity worth of food as they don't contribute as long as they're pillaged, then over time more from the lack of hearths. Villages provide +4 food up to 199 hearths, +8 from 200 to 499, and +12 for 500+. Villages also produce 50%, 100% and 150% as many goods for those same values.
I think the big issue is that the starting prosperity isn't calibrated for these numbers. You're probably looking at, in the long term, settling in at a prosperity below 5000 typically, maybe even more like 3000 if you maintain your hearths, but so many settlements start at much higher prosperity than is natural. So it feels like a 'downward spiral' just because the system isn't built to be stable at high numbers. On the other hand, if settlements started at 1500 prosperity they would all seem to climb. This is especially the case in the provinces and for settlements that have fewer bound villages. The thing is, it is set up so that they hit a bit of an equilibrium, it's just that this equilibrium is generally lower than the settlements are started with.