What i've resorted to doing is just using console commands to "cheat" my story along. If my head cannon dictates that the lord should join me, then I just bring them over. It's kind of a cheap workaround, but I'd rather have an interesting story than fight with early access mechanics personally.