I think that the main reason for that feeling is that there is something in Bannerlord that is fundamentally flawed: combat. As you rightly say Warband is, at the end of the day, an extension of the arena free for all fighting. You just do that outside, in open field battles or sieges, wearing armor and using different weapons, but it's the exact same game mechanic. And it works because the combat itself is really fun, I honestly can spend hours just fighting in the arena or at the training camp (in fact I spent an hour this evening raising Nord Warriors in native

). And that was in my opinion the genius of the first Mount and Blade, it built incrementally upon a combat mechanic that was incredibly satisfying.
Bannerlord... Not as much fun, I would argue. Blocking is wonky, hit boxes are unreliable, some animations are awkward. Combat is OK at the end of the day, but it's really not as good as it was in Warband. I honestly can't quite put my finger into what the problem is, but combat in Bannerlord when compared to Warband just feels clunky. I think many of the people who spent countless hours playing Warband and also tried Bannerlord will be able to relate to that.
At first I thought I just needed to get used to it, but after putting 150 something hours in Bannerlord I think I can say with confidence that it's not just that. Sure, now I can reliably (kind of) hit units running away from horseback, instead of flailing at them like a T Rex, and I can duel just fine in tournaments, but it's not as fun as doing the same exact actions in Warband. And if combat itself is not as fun as in the first game... Well maybe there's a need to step back and reevaluate the fundamentals.