Just walk away. The other lords never bother to come, so you shouldn't either. One "deliver this letter" quest will offset the amount of relationship points you lose for not participating in military duty.
But you should definitely fight if he actually encounters some enemies. As long as you can stay standing, and kill at least 30 enemies, the dread lord is unbeatable. He can do necromancy, after all. His old army could get wiped out, and he'd end up with all new troops.