Caravans affect the prosperity of their home city and the price of trade goods everywhere they buy or sell.
Priests are a way to rapidly lose favor with one religion if you attack them, and a source for gaining some in your party (either by rescuing them from bandits or capturing and recruiting from prisoners). In your party they can be paid to boost morale of your men. In your baggage train, enough of them will add to your surgery skill
Travelers I'm not aware of any effects.