Those axes aren't THAT heavy. It's just momentum that makes them powerful. Someone skilled to use an axe like that can easily outmaneuver a swordsman especially with their range advantage.
But yes, this needs to be addressed. To help without 'nerfing' the axes, I propose that they have two stances which the player can choose from by hitting the x key the same way throwing axes work.
The default mode would be how it is now. Only the axe should behave naturally. Swinging it like a baseball bat should make the attacks slow (maybe a tad bit slower than it is now), but powerful when they strike.
Hitting the x key makes the player grip the axe closer to the head (like one would hold a spear/polearm), making the axe faster, but weaker with a shorter reach.
To make up for this, shields need a tweak too. Top-tier shields should be made of metal (or reinforced with it) which would negate the "bonus against shields" effect that axes have, while letting the axe eat through the weaker wooden shields. (I believe Nords are the only faction with metal shields (Huscarl's shield), or at least that's what it sounds like.

)
I have the feeling this (my post) has been suggested before.
[Edit:] I didn't address the speed-issue correctly, the default stance should have a heavy momentum penalty giving the player a short recovery time that leaves them "stunned" when they miss with the axe.