After experimenting a bit I ended up with the bamboo spear. I prefer to try to take the rider down if I can, but about half the time the spear's aimed right it gets blocked (though if you get it right it's an instant kill). As mentioned above, it's much easier to go for the horse then finish the rider off after.
Spears are only really good for prolonged combat when the cavalry's charging at you before wheeling away and charging again, so you can use the impetus of the horse against it. When I've been the last guy standing against several desert riders fighting like that I've done a lot of damage with the spear. On the other hand, if they just charge and then melee from there on in then the spear's too weak (and, obviously, the longer the spear the more useless it is in cramped quarters). The tactics I've settled on are basically to stand in the front line and use a long spear in the first charge, dismounting as many riders as possible. Then go for the sword and keep focusing on dismounting horses in the melee, leaving the riders to my troops. Any cavalry that manages to wheel and charge again gets the spear once more.
In original M&B I used a two-handed sword to do the same thing. That gave you the option of having a decent range to take the rider out with a thrust, or to side-step the charge and slash down the flanks. In WB I prefer to avoid the one-hit-KO missiles though,, so I've gone for the sword, shield, and spear combo.