I think this is a good discussion for the game. I have three major issues, most of which have been mentioned here. One is that cavalry seem to not often use couched lances in the charge. Two is that the cavalry collide with each other in an unrealistic or unhistorical way (the entire point of training a cavalry unit was so they could ride in formation). Three is the recurrent problem of charging at a looter and my horse being killed. For me, it seems unrealistic. The reason has less to do with physics and more to do with realism.
It's not very easy to generalize about cavalry training, attacks, and tactics. From classical Hellenistic cavalry to medieval Norman cavalry is a big jump. While it's true that cavalry has always struggled against well-positioned, well-drilled infantry, such infantry formations were more the exception in medieval warfare (at least before the late 13th and 14th Centuries) than the rule. Against less-disciplined militia forces, heavy cavalry charges were very effective -- and among the chief reasons that they were so is that less-disciplined infantry tended to panic and flee when charged. That's really where I get frustrated with the Bannerlord mechanics -- why are looters and other low-tier infantry not only standing up to a charge, but counter-charging? I think it would be nice if the mechanics could be shifted so only higher-tier troops would do so.