Prosperity is negatively effecting food production for some dumb reason
This is certainly on purpose though. It's just a kind of heavy handed approach to preventing Prosperity from ballooning out of control or growing indefinitely. Having enough food in the stores allows the town to maintain a garrison and allows Prosperity to grow unhindered. Properity gives a penalty to Food Production to simulate "mouths" to feed. When Prosperity is low, then the penalty to Food Production is low and total Food Production stays positive, adding to the stores. The "Prosperity" penalty to food production gets bigger and bigger as Prosperity goes higher and higher (linearly I think). At a tipping point, which is where the "Prosperity" penalty is larger than the total Food Production, the Food Production drops to a negative, signifying that there are
too many mouths to feed and not enough food to go around. Food stores drop to zero, and when they do the town gets a big negative modifier to Prosperity until overall Properity drops to a level that the "Prosperity" penalty is no longer greater than the total Food Production. Then the negative modifier disappears and things reverse. Total Food Production becomes positive again and the food stores begin to accumulate until the "tipping point" is reached again and the cycle repeats.
The problem is that Prosperity grows to it's limit again too quickly, not allowing food stores to properly build up and in effect the town is perpetually starving, destroying the garrison and making it a sitting duck for enterprising young lords to come conquer.
You can add food to stores by selling grain to the town, but it gets used up quickly or purchased by caravans. Even if you add thousands of grain, the Prosperity will just grow higher and make it harder to sustain the amount of food stores needed to prevent the garrison from starving, eventually crashing harder than before.
When the tipping point is reached Prosperity needs to level out so an equilibrium is reached and the town has ample food stores for siege purposes, and isn't oscillating between starving and not starving all the time.
I hope I explained that well enough. I think I made it a bit more complicated than it needs to be. But trust me, they have it set up that way as a "release valve" for Prosperity so it doesn't grow forever. It's not to allow the AI to cheat and win sieges, and the AI has to deal with it too; you can tell because Prosperity levels for their towns will stagnate at around 6-7k if you look in the Kingdom menu.