And if they are OP for what they are supposed to be it's probably because there really isn't as much of a difference as there should be between "peasants" and "elite" troops in this game.
^^
This is 100% the issue.
In Warband, the difference between bottom tier troops (recruits, tribesmen, looters) and elite units (Tier 5's: Knights, Huscarls, Sharpshooters, etc.) was palpable. The disparity between them was extremely distinguished and noticeable. Even the gap between Tier 2 and Tier 4 units was quite profound.
This was good because it really gave the game a feeling of variety and highlighted the major differences between the level of "professionalism" of various troops. Also, it was good for the player, because it gave them incentive to try and upgrade their army as much as possible and provided the feeling of satisfaction upon seeing their force grow and develop and succeed.
In Bannerlord, I feel like there is just too much parity between the trash-tier units and good solid troops. When it's all said and done, there's not much of a difference between them.
The factors behind the combat-efficiency-gap in Warband were easily understood:
-Metrics like Weapon Proficiency, Power Strike, and Power Draw played a huge part in determining how effective a unit was going to be in combat. Units with low Power Strike or Weapon Proficiency were going to get bullied -no matter what.
-Also, equipment -especially armor, was crucial and tied directly to how well a certain troop would perform on the battlefield.
-Lower tier units were also generally equipped with less potent weapons, be it melee weapons or bows.
Armor worked well for the most part in Warband -it wasn't perfect by any means, but high-quality gear could protect the units (or the player) wearing it fairly well. That being said,
armor in Bannerlord is just janked. It doesn't protect nearly as much as it should and this is very disappointing. Now no one expects to be an invincible juggernaut in this game -no matter what armor they're wearing, but I think the whole "looters throwing rocks" complaint defines the issue perfectly, and this is why looters (perhaps unjustifiably) get so much controversy and hate.
I'll admit, it's frustrating when these starving, flea-ridden, scruffy men in rags take down a unit (or player) wearing middle-upper tier armor using just their rocks or crude weaponry they scrounged up while picking through the fields. But we must stop and ask ourselves, is it the looters' fault? Or the Bannerlord equipment/armor system?