For me personally, after 97 hours I have been in one siege defense. To me it's hardest to be a part of it simply because of the cost of entering the settlement after the siege has started, more often than not I'll arrive there after they're already besieging it, and it just results in me fighting to the very end (even if I'm outnumbered and WILL lose) outside of the walls. If I can halve their numbers they wont have the numbers to take the castle which is worth it imo.
The one time I did do a siege however, I was defending a siege of 600 men with 200 men. On realistic. I should have lost this no? I did not. I had a one-shot bow at this point and with the infinite amount of arrows the siege tower on the left never even made it to the castle (seen as I kept killing the guys pushing). At one point it glitched out and they stopped getting more people to push it, so I just stood there 1 shotting the group, 1 shotting the reinforcements. They didn't even get into the castle, not with the other siege tower or with the battering ram (even without my input, although to be honest I don't know they even used these).
It was however a VERY good feeling, through my command we managed to take down an army that outnumbered us 3 to 1, and only lost like 30 dudes. It was a very good moment.
TLDR:
Make it cheaper to break into the castle when a siege is going on, because siege defense is cool, why would I break in and lose halve my army if I can also just fight the enemy outside and guarantee they won't even be able to start the siege?