Without a single doubt for me it's the UI. This will sound arrogant, but I'm also fairly certain that for a great number of people that think that their biggest gripes are with other things, the actual lack of enjoyment will stem from the grinding issues that build up during a playthrough due to a staggeringly high number of weird decisions on how the UI works.
It's also the fact that I'm pretty certain that a lack of features, bugs, optimization or balancing will be sorted out by the end of the early access or further development, and gradually many, many things will be added to the game, just alone from player feedback. But let's be real: UI design isn't sexy. You don't really look forward to something that just limits how annoyed and bored you get during a game. What anyone really wants is content and cool mechanics.
I mean, Bannerlord definitely IS an improvement from Warband in many, many areas. But we're talking about a game that had to live up to the player convenience standards of people who knew that this was a grand and ambitious project made by a handful of people. In such a case the (sorry for saying it like this) godawful, but at least working, UI was good enough. Now we're talking about a relatively big commercial project that has inherited a lot of the relics from that godawful UI, impacting a large part of the gameplay loop on every station with things that (at least to the layman) seem incredibly easy to fix.