Eh? I don't think you can upgrade a village to a castle or city in vanilla Native. Anyways, Prosperity can be gained 3 ways:
1) Upgrades such as the Mill in villages will upgrade the prosperity slightly.
2) Caravans. In citys, caravans will transport good and money from city to city, increasing prosperity. Obviously more complex than that though.
3) Time. Villages and citys will increase in prosperity over time. Try to stop your village from being looted to see an increase in prosperity. It takes a while though.