What are those friends doing? Are they waiting near you, riding around chasing 2 farmers? Is it taking you a long time to build the Ladders/Tower so they got bored and left?
I just finished capturing a castle, playing as an Independent Kingdom (I'm the Marshall of course), had 2 vassal armies set to Follow me. I waited until they finally caught up to me at the Castle, then started the Siege. A companion with high Engineering made the Siege Ladders (thank goodness) only take 6 hours to build and my vassals never moved away. When I moved in to take the castle, they both joined in.