I've been playing for about 100 days.
Almost all lords (all from 2) of my faction have less than 20 units, are permanently captured by bandits and can't build up anymore.
This leads to the fact that all other factions can take all fiefs without any problems.