I think the problem you've identified is due to the way territory often changes hands back and forth, over and over. Or it trades between multiple settlements.
So a successful siege doesn't get you much. Because your new castle/city starts off with 50 defenders and every ai party on the map knows this immediately and begins to move to come take it.
So I just think the problem here is so much more significant, and shows how really empty this game is inside.
There really needs to be a cycle of war and peace, conquest and consolidation. However there is literally nothing for the player to do during peace time, so peace is boring. As well the game economy is really divorced from reality, because it's more profitable to be at war than it is to be at peace
So the game needs moments of peace, to add weight and significance to the results of wars, however there are zero systems that exist that make peace interesting at all.