Cities prosper because of trade, and caravans go to your town if it's safe (i.e. caravans and farmers don't get attacked nearby, while travelling there).
Destroy any nearby bandit lair. Being at peace greatly help increase prosperity of a town. However, it won't become very prosper until some things are tweaked.