It's certainly the intention, but with the current implementation fiefs don't gravitate towards a prosperity they can reliably supply, but rather exhibit a suicide sinusoidal prosperity oscillation that kills some fraction of your garrison every time the food dips to zero. That makes it look like a mistake on the player's behalf, which it really isn't.
I don't think fiefs are made to gravitate to some prosperity on their own, I think food shortages are meant to limit prosperity to the level local economy can support at any given moment -all factors including. In other words, food shortages are inevitable and they are not meant to be avoided.
Now I am not defending this system. I am not saying it's good ...or bad for that matter. It's very complex one and I don't understand it in all it's complexity. I am simply pointing out that fighting it and trying to prevent food shortages is like pissing against the wind. Since economy is designed to autobalance, when you as a player do something that upsets that balance, usually by trying to raise prosperity and food supply, the opposite forces gets triggered to restore the balance. More effort you make, the stronger opposite force becomes.
This is why players gets frustrated, because it seems, that game itself plays against them and their efforts. They see food shortage, go buy ton of food, dump it on the strawing town ...only to see prices crash and caravans from all over Calradia rush in to buy it off -carrying it back to places where you bought it from to begin with -because by buying all food there, you have raised the price and thus profit in bringing it back.
So my advice is: don't dwell too much on the "housekeeping" your fiefs. Build improvements, set policies, but consider fiefs only as a resource. Don't try simvilage, simcastle or simcity on them. It won't work.
There of course is an issue with strawing garrisons and judging from the fact, that devs are tweaking it, I take that they are aware of the problem. But here again overall intention is to prevent garrisons to become too powerful and unbalancing the game. I think solution would be to create some buffer that would prevent food shortage spikes causing garrisons to desert. As you have pointed out, granaries right now does not do that.