All caravans have owners, so it's practically the same.
The best way is to attack a lord, because if you fight some of them in future, they will hate you any way. And also you will weaken the enemy, by beating him right at the start. Profit.
But any way the button needs to get alive.