I found this happens when I have other parties in the battle besides mine, so sieges generally and large battles. It seems like when re-inforcements come on, they come into groups all over the place and they're mixed in with everything, archers with cav and so forth. It's honestly horrible and makes trying to command at that point pointless.