I came across a party of looters who were fleeing from a bigger party of imperials. I left clicked to engage it, but my party would instead follow it. I tried walking "into" the enemy party (with CTRL) before left clicking, but my party would instead back off the enemy party and proceed to follow it.