It'd be better if some of them went to your capitol (and for you to have a capitol) and wait to ask to join you (like in warband) and be dormant during this time. It's fine if they leave the game IMO but if the AI gets some of them for free, the player should have an equal chance of some defecting for free.