I've had a few defensive sieges. Often being completely overwhelemed because AI will not attack a town or castle they don't have a guaranteed success at taking.
However, the vast majority of issues is also that the AI build siege camps too quickly, and the only time the player is notified of a siege taking place is when the battle itself takes place. If the player were allowed a chance to turn up to a defensive siege more, and the battle took place at the castle (essentially pinning the sieging army in by the garrison + militia, and the player's army), it would make for better gameplay overall... rather than just taking to the field again.
I'm not going to lie though, I don't even really enjoy sieges. Attacking is boring, because its just choke point after choke point, defending only ever occurs rarely and is a cluster **** of the AI dominating because they refuse to siege a castle unless they can definitely take it.