Caravans are definitely OP. All caravans, not just player caravans, need a speed nerf so it's realistic to attack them, particularly with cavalry heavy war parties.
Also, one user pointed out that it would be cool if the caravans could be completely retooled to actually be a live party in your kingdom, not just an automatic thing. I think that's a fantastic idea, but let's not get lost on flights of fancy.
When bandits successfully raid a caravan, its loot should be stored in the nearest aligned bandit base as well (Appropriate balance-related decay and caps of course) giving more of a reason to attack them, and a greater reward for the 7 on 20+ scenario. I don't know how deep the bandit mechanics actually are, but if bandits themselves could have an economy function and become more dangerous/powerful as they successfully raid (IE, Mountain Bandits start spawning disproportionately more Highwaymen instead of Hillmen) creating a greater incentive to fight the bandits (or if you have master Roguery, better use of the Merry Men perk when it functions)