There are 3 factions in the game with cavalry (and of course bandits, manhunters and mercenaries have horses, and companions can have horses, oh and sword sisters too), these three are Khergits, Vaegir and Swadian.
How to get them?
Khergits are the most horse related faction in the game, even their recruits can have horses. And above recruit (called Tribesman) they all have horses.
Next come the Vaegir. To get horses with Vaegir you get a recruit, upgrade to Footman, then to Veteran (that may, or may not have a horse), then to Horseman and again to Knight. The Horseman and Knight always have horses.
Then to Swadians. To get horses you get a recruit, upgrade to militia (I think, if not its called footman, its only 1 way to upgrade a recruit, so its that one), upgrade to Infantry, then to Men-at-arm, and finally to Knight. The Men-at-arms and Knights always have horses.
To get bandits you could either free some imprisoned steppe bandits or mountain bandits, or free a looter and train him to mountain bandit.
To get manhunters you free a manhunter and upgrade him. Manhunters only have 1 way to upgrade for every level, so you always end up with cavalry.
If you want mercenary cavalry you could either hire them in taverns, or do it the hard way. Free a farmer, upgrade him to watchman, then Caravanguard, mercenary Horseman, and finally mercenary Cavalry. Caravanguard, Horseman and Cavalry always have horses.
Its a bit harder to get Sword Sisters, but its really not that difficult. First you need to free a peasant woman and upgrade her all the way. Its only 1 choice for every level here too. I think it go to camp follower, huntress, camp defender and then Sword Sister. Only the Sword Sister have horses.
Then lastly its the Companions, you find them randomly at taverns and you can give them equipment, so you could give them a horse.
I hope that was slightly helpfull : /