A horse should be able to make small adjustments in advance on its own to avoid an obstacle (the player recognizing those obstacles and turning the horse in advance will avoid unexpectedly having your aim spoiled). The horse should have a tendency to make larger changes in both speed and direction in the face of impending disaster or impact, whether the rider likes it or not. Such sudden changes by the horse should be reduced or made more gradual and further in advance with higher riding skill, and being able to make a lance charge or run down infantry without the horse freaking out should require at least moderate skill, or familiarity between horse and rider. If an impact does occur with a solid object, the horse should take damage, possibly fall, and the rider risk being thrown off violently (both fall chance and damage to horse and rider depending on the speed of the impact, and being thrown off if the horse stays upright depending on skill). If you screw up badly enough, you should suffer the consequences.
A fleeing horse should more-or-less maintain direction (with some small random variation) unless there's something in its way, in which case it should turn in advance slightly to avoid it. Once clear of impending threats (people or non-horse animals), the horse should randomly either slow to a walk (until it leaves the field) or stop.
As pointed out, having the horse crash into trees and rocks is unrealistic and NOT fun. Having the horse behave like a car (it does EXACTLY what you direct, until it can't and things go wrong) is unrealistic, and not much fun in my opinion. Besides, that also makes the horse a bit too overpowered too easily, because you gain the benefits of a combat mount with ANY horse. Granted, a highly skilled horseman on a large, powerful mount SHOULD be overpowered, but it should take a lot of work to reach that mark, not be something you gain just by paying the few hundred credits for some tame riding horse and devoting a single point of skill to riding. That riding horse should be EXTREMELY reluctant to charge into a line of men, and a bit skittish even about running into one lone guy, without proper training of both horse and rider.
Basically, I want cavalry in Bannerlord to BECOME as powerful as in original M&B, not be that powerful from the start, yet be easy to learn for a new player. That takes a dependence on character skills, and not so much dependence on player skill.