Lords usually don't like following you cause of too small parties. If you give'em some troops (up to 250-300) their usually always follow you until you decide to stop. Meanwhile, you can easily solo most sieges without any allies support - all you need is high surgery skill, and some good top tier troops. But there're several thing where at least one lord's help can be invaluable. First of all - you have no limits in giving troops to your lords - for example you can give 500-600 soldiers to one lord, and all of them gonna count if lord gonna help you with assault - so why do you need all these problems with gathering many lords when you can invite just one, but he'll be as strong as many?
Second thing - while besieging a fortress, enemy lords can easily leave it and escape, so you'll have to waste time to hunt down them later (especially if it's last enemy fortress). So when lord with huge army is with you, enemy lords likely stay where they are and gonna get wiped.
And most important thing - lords have no limits for prisoners. So when you're besieging town with 1500 soldiers (it usually happens with last town/castle of faction where all lords of this faction gather), and kill about 500 of them in battle, you'll get about 1000 prisoners in the end. And there's no way you can get all of them alone. Actually all prisoners and saved units you're not getting are going to your lord (or divided between them, if there's more then one lord). So since lords have no limits for prisoners - your lord get them all. All you have to do is to order him to go to your nearest castle (may be even those you just captured) - every time lord come to your castle of city it drops all his prisoners there. So if it's your own castle or town - you automatically getting all these prisoners which can be either sold right there or recruited by yourself.
So you can conquer the world with help of just one lord. Easy.