You shouldn't be. It's really the wrong system for a game like this. It means every character will play the same for the first 2/3 of the game. Every character starts out bad at leadership and engineering, and it's basically impossible to level either until mid-late game. By that time the story of your character is already mostly written. Every character will have high steward, because it's almost impossible not to. Every character will have either high riding or athletics, for the same reason. Every character will have low-mid scouting and medicine, again because they just level up passively. There is almost no differentiation between characters, they will all be the same.
I agree with you but think its not exclusively the leveling system at fault. Part of it is that in terms of content, there is only 1 content heavy way to play Bannerlord. Be a vassal, or found a kingdom, and paint the map your color. A skill system that differentiates on how you play only works if you can play the game differently. Now, in a game like M&B the framework for multiple playstyles is easily there - but almost always vassal/kingship is the only one with any meat to it.
That compounded with the fact that our starting choices aren't very sandboxy. We're always an inexperienced youth. Now, indulge me for a moment:
Imagine a Bannerlord where you could just play as a caravan mogul with the same content/satisfaction. Imagine you could start your character as a 50 year old man, well skilled in engineering, trade, and charm, but absolutely useless in combat. Imagine you could actually be enticed to join another lords party or a clan or kingdom because you generate money, or are useful army-support. In this hypothetical bannerlord then this type of skill-system, I think, would be fun.
However, when the game basically has one playstyle, and has very limited character creation (always young with a wide spread of low skills) it absolutely becomes exactly as you describe, the same for the first bit of the game with the same skills inevitably just leveling.
I think what many of us like about M&B is how the game organically creates a "story" based on our playthrough. I think the idea is to have a skill system that meshes with that, so our character organically becomes good at what they did in their "story." That idea currently fails in execution, I'd argue, for the reasons I gave but could work. Will the game ever flesh out enough to make that system work? I dunno, probably with mods maybe never vanilla. So would it be better to maybe just revert to the warband style? If we're unwilling to make the game world that makes the other system work, then yes.