There is a concept of strength in the game that's more than merely army count and when lords choose to engage or run away from you they do it based on strength assessment. Notice by the way that when you besiege a town or a castle, enemies will park outside as long as they're weaker than you, then once they're roughly the same strength they'll engage you.
I hope that other factors like moral difference, renown, etc, army composition and risk/reward (e.g. I might lose this battle but it'll be a Pyrrhic victory for my opponent who's in a weaker kingdom) can also come into play, though I'm not sure to what extent that is the case at the moment.