There is already a caravan ambush quest. It should only be logical that if your troops outnumber or outskill the bandits, they wont attack. The average player can live without battles for a full minute, this kind of forced conflict is really not needed.