Looking back at Warband it just seems silly now how you could get smarter by bashing heads in.
It's silly, but the alternative is to design the entire game so that the player doesn't intentionally do grindy stuff to get skills. It's something that requires total unity of game design, which taleworlds has never really been capable of.
I don't think it's just a question of numbers. I don't think warband's game loop (which is what bannerlord is basically) can support passive skill accumulation because it is inherently grindy by nature. You have hundreds of settlements which you capture the same way, infinitely respawning enemies and very little player skill involved along the way. There is no easy solution but I don't think just increasing the xp will do it.