I wouldn't even say that there is a unit to be feared most... I mean being one against a band of Dark Nights makes a tough fight (if you minimize the damage to yourself and make them dumb it's tolerable though
), but when there is a good 200+ warrior skirmish a particular troop does not make a great difference.
There are some basic rules, which one has to obey to slay overwhelming enemies from a horseback, one and the most important of them being to shield yourself with a damn heater at all times and watch your back carefully.
If you are charging a mounted lancer than you'd better change your lance for a sword (a morningstar will do better) and go to his left. Don't want to get a crouched lance blow. And don't charge a line of cavalrymen before your own cavalry comes out to play. You'll be surrounded in no time.
If you meet any infantry lancers as well as archers you better be cautious not to be open for a good frontal hit (or an arrow, as a matter of fact) or one from the side. The latter happens when you are especially fascinated by taking down some smart-arse halberdier, who just took down one of your heroes. And, of course, at all suitable times you should use a lance to stay away from witty guys with twohanders.
And the heater works well to charge archers. You just need to crouch after they make a shot and before you ride past 'em. Another way of doing it is to make sure that the archer aims at someone else.
And the last one... be quick enough to slay all the archers and cavalry before the enemy takes down all your troops. After that the battle would become a free target practice and I doubt if you could win one like that.
That all's pretty hard though, or, probably, I'm a lame player. Speaking of which, I've got completely no idea how one would do the same thing on foot, though I dismount myself from time to time to get done with archers on hilly terrains.