If it helps I think this might be happening when another army has chosen the castle I'm besieging as its next target.
During a war, the first two castles I besieged had this issue and later another army came by and set up camp to besiege it and I experienced this bug even before they showed up. The third castle was close to the second and I didn't experience the bug when the other army decided to go attack another army.