The town's description seems to complain about people being hungry and tools being broken, so in a recent game I started leaving piles of grain and tools in the town's shop and after a while the town become very prosperous. I'm also protecting my towns from raids so it might just be natural...