Khergits are a nightmare in large numbers in a pitched battle. Vaegir Marksmen are extremely deadly in a siege. I participated in a battle of each kind. Both times it was the Swadians against them.
In the open battle, though I cut down more Khergits than I could count, the Swadians took terrible losses (it was like watching the Persians against the forces of Leonidas before they were shown the secret route).
In the siege, the Swadian forces were shot mercilessly by Vaegir archers and marksmen. Their rapid firing and decent melee capability meant that the Swadians were left utterly defeated. As before, I slaughtered a great number of Vaegirs of all kinds, before I was cornered and defeated.
If you never get close, I'd say the Khergits are deadlier, but if you get close up, they're fairly easy to slay. The Vaegirs are a little more challenging in melee.
Khergit horse archers are worse than any other archer, however, for the simple reason that your troops will find them difficult to surround and defeat, due to their high speed and maneuverability. So my vote is for the "Horse Lords" of Mount and Blade - the Khergits.