As others have said, the lack of impact & damage for both parties, as well as morale/fear/nerve, makes it hard to get this right in game. Its not just the physics of pitchfork versus knight, but also the psychology in the situation. I don't think we'll be able to perfect it, but I'd prefer a little more oompfh for both sides. Most mods I've seen thus far increase charge damage but don't increase fall damage, for example.
Also, this is unlikely to ever be modeled in game, but my understanding of a big danger for cavalry throughout history (any of you nerds wanna correct me, go for it) was your horse falling ON you if you both went down. If a horse lands on you, at best, you're staying down for a little bit and at worst you're smushed.
Right now when my horse gets taken out I do a 20foot dab and roll and then hop back up and slaughter like 5-6 archers.
I don't thinking about a 1v1 knight versus peasant really captures the missing dynamics, to be frank.