You don't realize because it is not true.
You gain focus points and attribute points faster at the start because you need less skill levels per level but the speed of learning doesn't care if you have 300 stats or 1 it cares about what the actual level of the skill is, the attribute points and the focus points.
However if you want to level charm fast just buy a horse from the lords you pass buy and it will explode.
Actually, I went back to level a character and noticed that at level 1 the experience gain was at a x10 whereas by level 10 it was about x6 - so there is a climbing exp grind based on what level you are as well as your focus points. In practice, this doesn't stop you from leveling skills you want to. Rather, it just makes it take longer. However, I would say the vast amount of exp required and the slow trickle for such, rather than the multipliers, are the real impediment.
I was also quite aware of how easy it was to level charm. I don't even buy a horse, I just give each passing lord I see 100 gold and call it a day - works just the same. If I like the lord, I might throw in a spare sumpter horse.