I have experienced this a few times now, whenever an enemy sieges a castle the defending faction doesn't intervene. Instead they stand around the siege camp and keeps "chasing" the enemy army. Even though they easily outnumber the enemy with castle defenders.The only chance that they break siege, is if the enemy routs due to a large army of defenders waiting outside the castle
