One of the most important parts of any siege is to clear the ladder top. Like Lord Shields said, if you let your troops sit at the top of the ladder they'll be arrowed to death and then the lead ones will be hacked away. Swadians Knights are especially bad for this, they are tanks and when they take out a defender they are just too slow to advance and gain ground, defenders come back in to fill the spot and more of your knights will die over time.