Their personality also affect this, as does your relation to them. You might have low relation with a lord and see him join you because he is either good hearted or military personality. Others will take more relation before they consider joining you. Try and have at least 25 relation with them and you should have a good showing.
In the last days of my previous kingdom I had everything except Suno, Praven and the north (Nords / Vaegirs region). The only remaining faction were the Nords. I had 16 lords almost all of them 95 to 100 relation. What I did was start a feast, wait for everyone to show up. When the time was right I called a campaign, and took my super 2400 men army on a stroll. After 30 days (everyone leave after 30 days no matter what) of total carnage I had taken 20 castles / towns from them and they only had 1 city and 1 castle left! Having good relation with you lords is quite useful!
I remember the battle of Thir. I had taken the city, was laying siege to the castle next to it (took 2 days, siege tower required). While I was taking it the Nords showed up at the doorstep of Thir and just started the siege. They had a massive amount of lords and troop. I ended up bringing my whole army close enough for them to join in battle and I engaged. The battle lasted forever, they had 2400ish against my 2100ish. Quite the epic battle! After that the Nord's back was broken and I just steamrolled them.