you will never face the whole 1k+ elites force, maybe not even half of it, then the siege will move to inner city/castle battles which win themselves if you have companions properly geared/leveled
in short no matter how many troops the castles have, you will only actually face 300/400 ish of them, letting your castles go and conquer them back is MUCH more cost/troop efficient than facing lord armies on the field or defending, while the whole lords army is busy with one of your towns you can conquer 2-3 of theirs