Yeah I've seen this a fair bit in my game as well. I remember a couple patches back there was a fix that addressed the issue and said it should be better now. I think I have noticed since then that it does happen just a little bit less now, but yeah I'm still seeing it commonly enough.
I think part of the issue might be the snowball-y effect of armies. They siege/defend 3 targets in a row, and then suddenly they are far away from home as 6 parties split up and each have like single digit troops left with a lot of wounded and prisoners slowing them down. Since bandit/looter spawn rates are quite high in most people's games right now it then just becomes a matter of time before they get caught up.