Take out the forrest bandit lair. This will help caravans reach Dhirim.
The lord of Uxhall gives quest to take out bandit lair. This quest gives lots of exp (3000?)for you and companions. Guild masters in Dhirim and Uxhall also gives quests about forrest bandit lair. (just wait by the lair if your looking for someone special)
The lair is usually in the forrest north of Uxhall, but sometimes north of Dhirim. (along the east-west treeline between Knudarr and Tilbaut castle) Sometimes it spawns elsewhere, but it should not be to hard to find. Always in a forrest close to either Dhirim or Uxhall.
If your up hunting seariders, talk to lord of Rivacheg to get quest to find sea riders landing. Its even easier to find (always at the coast) and its the fastest way to level up companions.
I suspect city prosperity doesnt matter too much. In my game, Dhirim goes between poor and average, but it brings in lots of money from rents and tariffs. Im at peace and gets lots of caravans coming to Dhirim. The constant feasting in Dhirim might affect the rents, but I think its all the caravans that gives me high tariffs. Wich could be because there always lots of lords around from all the feasting to protect caravans.