Once rich, other cities will take their goods else where since your price indexes are so low (less than 1000) meaning they can't get a good price for whatever they're selling. So your city will move back to average.
Well normally, it's caused by raiding parties (bandits/looters/etc.) which kill/capture your farmers. They have to reach bigger towns AND come back in order to generate wealth.
I always wondered why my city was pending between poor and average for some time. When I searched the surrounding area, I saw immense numbers of bandits scouting the area, because the game had spawned two bandit lairs very close to each other
I cleared them out, and my village is rich ever since then.