Factions get too many lords if they have only one settlement. Look at this picture. I have prisoned at least 15 and they still have this many lords and these are not all of them. I get that this prevents the early demise of one faction but it also is preventing to kill other factions in the late game.
I gathered more than half of my factions lords and couldn't get 1 city.
I gathered more than half of my factions lords and couldn't get 1 city.