Fraction? XD Faction...
Anyway, what I think is happening is that all the lords of the factions that have been wiped out are deserting and entering in your faction... So, your king has a wave of lords without lands. Since you have a fief, they are chosen before you.
But, even taking that in count... Its strange... How much renown do you have? How many troops can you take with you? I think that, maybe, these two factors have something to do with the problem...