Khergits require different tactics, that's all. Separate groupings for horse archers, and different tactics in both siege and open field.
Everything in the Khergit army has a ranged weapon save the lancers. Everything also trains fast, moves fast, and costs very little. When sieging, just do the massed archer hit and run tactic - kill all of the defenders you can by holding position outside of the castle, then retreat until there are so few that you can send some lancers + companions in to cleanup, or they'll voluntarily vacate.
There's a thread dedicated to using Khergits in the open field, but the gist of it was that in vanilla native, just send your lancers to the left flank, horse archers, horsemen, skirmishers, and VHA's on the right following you until you're close and they are using their ranged weaponry, then tell your lancers to charge followed by telling the other group to charge. Basically what ends up happening is that the enemy heavy cavalry gets skewered by lancers, and your other group spreads out the ground enemy into easy pickings for the long haul.
I should also say that in the early game where you aren't the marshal and cannot control allied forces on the battlefield, Khergit's lack of tactics (they just charge) is often the least debilitating A.I. on the battlefield.
Rhodoks, on the other hand, are difficult to train, slow, and have recently been inexplicably made worse by their equipment that simply doesn't work with the A.I. I've tried a lot of ways to get the best effect out of their infantry under sergeants, and no matter what I do, they always under perform (unless I edit their equipment, which I loath doing). You pretty much need to be high level with high level companions and a ton of training to get anything redeeming out of them unless you're working with something like an entire force of crossbowmen. Still, if you do have the prerequisite amount of training to quickly turn spearmen into sergeants and some good companions, Rhodoks can start an unstoppable steamroll.
So my vote for the most difficult faction would be Sarranid (as of the latest patch) with a balanced cavalry/infantry/archer force. They are 'jacks of all trade' kind of like the Vaegirs, but have absolutely horrid infantry, light armor all around (though now it that it weighs more, offers no advantages), and archers that are as expensive to train/maintain as Vaegirs, but notably less effective. Actually, that describes pretty much every Sarranid unit in the roster: Expensive to train and maintain, notably less effective than an equal tier unit in another faction.
At least when you play as Vaegirs, you always know you potentially have the best archers in the game available to you. I really think the only way to get anything redeeming out of the Sarranids is to just mass mamelukes and master archers - which violates my idea of a 'balanced army.' Even when you do end up doing that (my opinion) cheesy tactic, Vaegirs do it better! And if you break down and press the easy mode button that is an all cavalry army, know that if you had gone with Swadian Knights instead of Mamelukes, you'd have a better force.
Sarranids: An army of over-trained, overpaid, underachievers.