I THINK that if they've been defeated you need to check in on them, other add to the party or maybe just visit the fief they're in. You'll get a announcement "you hear so n so is in this fief" or something. As others said setting them governor will also retrieve and activate them. I think if you've left them in a fief you can still start a party remotely.