1st of all - they appear only between dusk and dawn(including). It's possible to be light, light in dusk case.
I also think they attack only when you're injured(might be wrong, but that's how it always happened to me). I've lost 10k denars because they caught me with 5 hp and a lucky arrow got in; was irrelevant, since what they take from you is % - at that time I had over 300k, so losing 10k didn't matter. But anyway, sometimes you'll lose, that's for sure. At least on max. settings. If you have close to 0 hp, 1 lucky hit and that's it.
what you do:
1st thing, you're in the menu and you notice a small lag from loading. Immediately, prepare to mouse wheel down to get the shield out. Then mouse wheel up to get your sword out.
you're a cavalryman, so probably in heavy armor. Athletics help, no doubt, but you won't outrun them by much given the ton of metal you're carrying. Also, athletics don't work when you backpedal(you do it same speed, regardless of athletics).
Backpedal to the 1st corner, so you get rid of the archers. I use bastard sword, but that's a matter of preference. Anyway, all I do is side swings; no thrusts(which btw. - works crappy when you backpedal). Trying usually for a left to right swing, so I hit the shield less.
Avoid higher ground - I use a board shield, but if you're somewhere above them, they'll "hit your legs" or something like that. Try to stay on even ground. Keep your shield up and swing when you have the chance, but be patient - don't push your luck, the shield probably will hold, so stay cool.
Be careful not to mouse wheel up to take up your jousting lance instead of the sword; it ain't groovy when on foot
Keep the sword/whatever 1hander in the 1st slot of the inventory.
Find a newly spawned group of 6-8 bandits somewhere, save, keep reloading, set your troops on hold, go and try to fight them on foot for some practice.
Contrary what most say here, I consider the wall to be my friend and backpedal too. Just make sure you're careful and don't allow them to surround you(quick jump or run through them, even eating a hit, but don't get surrounded). Even with manual blocking, a board shield won't let anything getting through as long as you're in a corner and they can't surround you,
but be careful and patient and swing only when you have the chance. After you dealt with the melee, go in the face of the archers and force them to melee you. If you have a bow, you can take them at the distance since you probably aim much better/have higher stats so do much more dmg/have higher armor so you can soak up while they can't.