Losing prosperity because of lack of food, lack of food because of low prosperity is the indicator there needs to be something between them to break that loop.
I think building mills or maybe extra farms in villages like you said would be a good way to solve the problem by creating extra food.
But also building bakeries or distilleries, instead of creating extra food from nowhere(they can create some food, but not as much as a farm), should help with the prosperity because they're high quality luxury food. But it should not necessarily solve the food problem, so your people should still be happy enough to break out of that loop but still have to deal with starvation in case of a siege.
Different consumables (not necessarily all consumables but the same should go for fabric, leatherworking or metalworking too) for different cultures would be pretty cool. Every faction should have their own thing, spices for aserai, pure-bred horses for khuzait(expensive but not statistically that good), metal plates for vlandia. They don't have to have special uses other than being exclusive to their own faction.