It sounds like they are taking the initiative away from you. There are several factors at work: passive block stun, stances, movement, in-swing rotation, swing direction, animations (which have large arcs), heavy attacks, and quick attacks.
Passive block stun: If you are holding block for a while, lets say as soon as you can tell where he is swinging, he will be able to attack again faster than if you were to active block.
Active block: If you block his attack at the last moment it will cause him to have a slightly higher "stun" duration - so he will have to wait a bit longer before attacking again. I call them timed blocks because it makes more sense to me but TaleWorlds seems to call them active blocks.
Movement: if he is running around you and you aren't reacting to that with new positioning he can either hit you first or get you to put up the wrong block while trying to panic turn at the last second.
Stances: stances effect swing speed, combo speed, combo animation in some cases, block speed (may be just a feeling, can't confirm for sure), and how much damage your shield takes while blocking. Movement speed while swinging used to be different depending upon stances, but they either removed this or reduced it to an amount that is unnoticeable.
In-swing rotation: You've probably noticed that, unlike warband, you can rotate a ridiculous amount during a swing.
Swing direction: Depending on stance you're going to get faster attacks from one side. By itself this shouldn't be stealing the initiative from you, but a lot of little parameters like this can add up. If i'm in right stance (left foot forward) my left swing will be faster than it is in the other stance.
Animations: Swing animations in Bannerlord are larger than they were in Warband. This is why you can hit people who are to your side, or with some weapons behind you. A factor that contributes to this is that attacks hardly ever bounce in Bannerlord. I can hit with the very beginning of my swing and do damage. Patch 1.5 seemed to lower the amount of already rare bounces. They really only happened as a result of player error (weapon choice and attack angle).
Heavy attacks: If you hold for even a moment it makes the attack slower and heavier. This does more damage depending on how long you hold.
Quick attacks: Tapping attack and not holding at all results in faster attacks.
Some more obscure parameters may be contributing as well, but I think these are the important ones.