I think Vaegir troops, or at least their actual warbands, are quite overpowered. Their archers are powerful, but understandable as that should be their faction strength.
It's mainly the hammer wielding troops that are so ridiculous relative to other factions. It certainly is not their faction weakness. I had a quick look at their I3-5 stats and their powerstrike combined with weapon speed and blunt damage gives them a very large advantage. Playing as the nords, which supposedly have excellent infantry, and even I4 or I5 huscarl troops have a really tough time against equivalent (or even 'weaker') vaegirs troops.
Still, main imbalance are the bloody C6 Vaegir Elitniy Druzhinniks. Every army has a fair amount of them, and they can absolutely decimate anything else. No other faction troop gives me as much trouble as these guys.
I have 60 armour on both head and body, and 70hp. These badasses regularly knock me out in one or two hits, and that's not even when they are charging on horse for speed bonus or bump-slash. So imagine how well they do against regular AI troops. Combine this one-hit attack power with their heavy armour, and they could easily take out any other troop. They can be managed on the battlefield through positioning to stop their charges (e.g. hills), but the main problem is the sieges.
The amount of sieges I've lost just because of these guys is ridiculous. Sometimes I would see these guys would on average get 10-15+ knockouts each in a siege. Was tempted to record it in fraps or something to show how crazy they can be.
I find Sarranid and Swadian cavalry (or armies in general) to be far easier to deal with.