It really doesn't hinder gameplay in my personal experience, if anything it turns an obvious choice into one that has consequences and makes you consider if it truly is the right choice (do you have enough for a field battle or do you need the siege defender bonus to win?).
If you prioritize 25% (which can be lowered by the 125 tactic skill) of your troops more than gaining the advantage of defenders, so be it but thats your choice to risk losing the castle. I've had plenty of siege defenses because I value my fiefs much more than 25% of the troops in my roster and the ability to take on armies twice my size. Honestly if some of yall havent played recently, it is extremely fast and easy to recruit and rebuild your ranks these days.
I do have a question for people who claim they have played hundreds of hours with no siege defense. What did you do instead? Just let your castle be taken or did you fight in the field instead to not lose the troops to the break loses? What did you do when the sieging force vastly outnumbered your troops (assuming a field fight would result in a loss)?
The problem isn't necessarily the losing of 25% of your troops it is that it just doesn't make sense to do so. If you wait and allow the enemy to capture the fief you can just as easily without as much risk retake the fief after they leave due to the small amount of troop they leave as a garrison.
Also there is the psychological impact of just throwing away 25% of your troops. It doesn't matter how fast you can recruit more and build back up it is the fact that your losing so many men without any return. I mean in a field battle, a 25% killed because if you break in, they aren't just wounded, they are lost for good, is a pretty significant loss but I would also be taking quite a few enemy with me in the process of losing 25% of my force killed. In a break in, your men just "die" and the enemy takes zero losses in the process. If your breaking in and losing men that means your fighting and if you fighting that means the enemy should be losing me too.
This goes back to my first point, if I am going to suffer 25% killed in battle, I would rather do it re-taking the settlement and taking the 100 or so troop the enemy left as a garrison with me because that garrison actually reduced the number of their army.
Also you have to consider that not everyone plays with full damage on their troops. For someone who doesn't like I often do, a 25% killed causality rate is a pretty stiff price to pay. I would also say this is the same for a person who has a high medicine skill because most of their losses are wounded, not killed even in a large battle so a 25% killed for no gain just doesn't sit well.
I for one would prefer a small field battle being triggered where you must actually engage and overwhelm a portion of the enemy force before being allowed in the castle. Maybe give them field defenses so it would be almost like a mini siege but once you won the battle, if you did, you could enter the castle with your survivors which could be either more or less than 75% of your force and might be mostly wounded even if higher than 75%. At the same time the losses the enemy takes are subtracted from the sieging army. This at least isn't throw away 25% of your army for no reason.