Keep fights are set up to favor attackers
Sieges in general seem to favor attackers. I cannot remember that last time I lost an offensive siege, less numbers, trash troops, no siege equipment? Not a problem. It's like a field battle but I don't have to do anything, I just win.
The issue is that using ladders to attack the walls is very effective, most troops get to the top of the wall unassailed by missiles, and then just plop down on the wall in front of defenders fairly easily. In Warband the amount of troops lost to missile fire at that stage was usually quite high, and upon reaching the top, your troops actually setting foot on the wall was a hotly contested IF statement and most got brained as soon as their head poked up over the ladder.
If a few players are to experience a particular feature rarely, it becomes less worthwhile to pursue than a feature that more players are likely to experience often - all else being equal.
It is only "rarely" because of other imbalances in the game, mainly that it is easy, and you rarely need to be on the defensive.
If there was more reason or even an imperative for us to sit in castles/settlements we would have more opportunities to take part in siege defenses. A resting/fatigue system like the example from the
I'm Tired mod could help with this. In addition to lowering troop wages when in a settlement like in Warband, but instead tie it to how many hours rested in a settlement that day rather than a boolean
'were you in a settlement when the wage tick happened' which was easily exploitable by the player back then.
If you are wondering how the AI will handle a fatigue system and that it would take too long, or it would be too complicated, just make it player only, the game is too easy to keep giving the AI the same hurdles as the player. And the ImTired mod actually heavily modifies the AI to use the system; so it can be done, and done by one person, and inside a short time window.