✌The problem is when a castle is captured, it is automatically given to a
leader of a faction and is considered his property until he's given it away via vote/leader's will. And if you have
bad relations with him, you won't be able to enter a fief. I think it's the main reason, why it is done so.
So at the moment if you would implement appearing immediately in castle walls, you should also change the current relationship system. Not to do so a suggestion below.
Possible Solution
How it must be done instead to fit in the current system and it also be 99% the way you wanted from the start.
Attacker should appear in front of the conquered fief's gates. More specific:
The place you appear on the map when you leave a fief.
And only then if the attacker has good relationship with faction leader he can immediately enter without wasting a single moment of in-game time. If he can't enter, than OK, it's already an another story.
+ to this thread, because I fully agree!
No logic in attacker's party/army appearing so far away from the place after a successful siege.