I never liked it from the start. They put way too many brakes on leveling at the same time. Learning rate goes down when you go over the learning limit, which you'd expect, but at the same time, for each skill point you learn, the next one takes more xp, and for each level, the next one takes more xp. And on top of all that your overall learning rate goes down as your level goes up. They all combine to make leveling feel painfully slow.
At least in Warband, you knew if you killed a Swadian Knight you were gonna get 288xp, no matter what. Could you imagine if in Warband, as the xp requirement got exponentially higher with each level, the xp awarded per kill went down? So like at level 40 when you need around a million xp to get to level 41, killing a Swadian Knight now only earns you 20xp instead of 288 at the beginning of the game. That's kind of how Bannerlord is now.
But given that the skill system is what it is, your main mistake was in leveling smithing first. Its all too easy to shoot up 15-20 levels right off the bat if you happen to unlock a good component early on, and then that slows down your development of all your other skills. Its better to let companions do all the smithing since they hardly level much anyway.