In the mercenaries mod for Warband you could only open gates (doors too) if opponents are not within the vicinity, I feel that would provide a good balance for Bannerlord in allowing the likes of cavalry to go on the offense at the beginning of a round then defensive later on.