I think it have to be at least little amount of damage when people fell into ground from horseback. I think will be more realistic with this dynamic. The damage can be depend upon velocity of horse. If we consider horse is pretty fast and falling into ground with that velocity must be hurt for the rider.