After reading your post, I went ahead and looked at my game and I was suffering from the same penalty! So i tried an experiment and so far it has worked. I moved and aligned all my cav, Infantry, and ranged to be together in the party screen (including my companions that I flagged as cav) after a day it got rid of the penalty. I do not know if this is how it is but it worked so far. Ill post later if it comes back.
Edit: Nope, it comes back occasionally, no clue :O