Just so there's no confusion on the point, let me spell out the facts. I am correct.
Weapon Mastery makes it easier and faster for your character to gain weapon proficiency levels. The proficiency level that is "purchased" through points in Weapon Mastery is the maximum that you can you spend weapon proficiency points up to. So if you have a level of Weapon Mastery that says your proficiences can go to 140, then you can spend weapon proficiency points up to 140, but not over. Skills also increase more rapidly with use, IF, the profiency is below the limit set by Weapon Mastery. So if your Mastery max is 140, you can increase your proficiency from 139 to 140 with a lot fewer stabs than it would take to go from 140 to 141 . At higher proficiency levels, weapon proficiency points are pretty much useless since it takes several points to gain even one actual level of proficiency; thus practice is the easier and preferable way to gain these levels.
Personally, my opinion is that it's a waste of skill points to buy Weapon Mastery over 2 points if your focus is on "easy" skills (like crossbow use, or lancing from horseback), or 4 if you focus on the others. Beyond that point, the advantage of higher proficiency isn't worth the skill points needed for higher Weapon Mastery.
Your attacking speed is a function of the weapon proficiency level, NOT Weapon Mastery. Agility is taken into account for attack speed, which is why high Weapon Mastery may make it seem like that's what it does (since you require high Agility for high Weapon Mastery).
I further disagree that it's impossible to play on foot versus mounted enemies: get plenty of allies, use a polearm, and find a suitable place to hang out. Sergeants and Guards trump Knights any day of the week if the terrain is favorable to infantry. The horse-riders bunch up and get in eachother's way while the ground-pounders hack them to bits.