To get off the big bumps that knock players down for a long time you have to be moving relatively fast. At this speed the multiplier for damage on your horse is enough to dismount you in 2-3 head shots depending on the weapon used and the horse armor equipped. Couple that with the likelihood of taking melee damage or getting stopped by a spear and you'll quickly realize how fast that big tanky horse will actually go down. It's a risk/reward ratio that is balanced fairly well imo. As for maneuverability, if you take away the cavs ability to quickly move in and out of fights its just a big expensive useless chungus that gets snagged on barriers and shot to death.
I need some more playtime comparing the swing speed comparison between inf and cav, but I assume you're talking about the lance thrusts vs. inf spear thrusts?