When I was killing all the lords from the southern empire most of their remaining lords ended up defecting the faction with their cities, and those that didn't simply had theirs besieged by other factions since they were pretty weak.
I don't remember seeing a single city/fort that didn't belong to someone even after I wiped all of them out