I'm surprised you haven't had any incidents.
Admittedly it's rare, and if I do partake in a siege defense I usually end up breaking in.
Really when you think about it, you don't want to be stuck in Castle/Town as you've got no where to run. If you can't fend off the attackers you're basically screwed. I think even in most historic examples defenders were basically holding on til a friendly force could relieve them.
You're not missing much though. You might think being a defender, puts you at a big advantage. Unfortunately much like ladders in offense - the A.I. can't figure how to use defensive catapults half the time. They also fall off ledges and do all number of idiotic things. So yeah you might manage to fend of a larger enemy army because their troops only climb one ladder the whole time. But you'll loathe siege A.I. all the more.
The A.I. is always wish-washy about attacking, unfortunately I'm guessing that's a balancing thing. Because if A.I. Armies never abandoned sieges you'd have real snowballing issues.
What I don't get is why Sieges have to play out in such a scripted fashion. The A.I. just cannot seem to handle anything that "doesn't go according to plan".
It doesn't seem like it should be that hard to program. You literally just have 3 static points to attack/defend. What's strange is Archers have no problem going to sensible firing positions. But my lord, melee Troops may as well all be inebriated.
What I don't get is why formations don't focus on their objectives sensibly. The Ram/Gate formation should be entirely focused on breaking down the gates, even if the ram is destroyed. You know hack 'em down with your weapons. Likewise each Ladder/Tower formation should be entirely focused on their section and only abandon their objective if their equipment is destroyed.
I think Ladder/Tower formations need some kind of queue. As is to me it looks the A.I. is trying to user ladders as a "path" to the enemy (Which is fine for ramps, but not ladders.) Which is why they get stuck so much. I mean surely units in these formations must have some sort of number assigned, otherwise how is that I'm able to transfer them and how are they spawned in with larger battles? Why can't Unit #1 go up the Right Ladder, while Unit #2 goes up the Left Ladder, then #3 goes up Right, etc. etc. etc. And the only reason Units shouldn't be going up the ladders is if their morale breaks.
I get there is some added complexity in that the Ladder formations need to defend themselves, but the way the maps are that really shouldn't be an issue. Maybe Ladder formations need to be more "tone deaf". I dunno just seems like it shouldn't be that outstanding of an issue.