I agree with you on that, Lotus Dragon, Oskatat, Beserker Pride; and you are right, I think I have overlooked weapon proficiencies as far as single player goes (if weapon speeds increase with levels then there is no problem there). As for warband (i.e. multiplayer, no single player yet - that I know of - so I cannot comment); this is where I feel the hand to hand combat is lacking. You can't level up, and it feels as if you're stuck forever on the most basic weapon proficiency level.
You've drawn to my attention then that perhaps the problem is with the standard proficiency level of a player in warband (multiplayer, beta). It is surely too low. I would propose it be increased to make things feel and look more realistic and exciting.
So for multiplayer, shouldn't I expect to play as a soldier with at least some training rather than, in medieval terms, forever as a peasant; yet dressed in heavy armour and commanding an army; who acts like they have never held a sword before? As an example, consider even a complete novice lumberjack/timbersports person - they would know that if the aim is to chop a block into two in the shortest amount of time; and that it can be achieved in a matter of a few seconds, they shouldn't 'charge' their axe for 2 seconds by their side before each swing. Now give that block arms, legs, a brain, and a weapon and know that if you wait around like you have to in warband before your attack actually gets thrown, its going to hit you! Only in warband (multiplayer), it doesn't, because it's just as slow as you are.
Being able to kill a horse then the knight immediately afterwards sounds awesome, albeit a little extreme! I am hoping the weapon speeds get increased to more practical levels in warband multiplayer - at the moment those actions seem to take about 8-10 seconds... I've tried it a few times in the beta (killing a horse, then going to kill its knight), but I usually get stabbed in the back with my sword in the air waiting patiently for it to finally thrust towards the knight.