Charge mainly affects how much speed you lose when trampling infantry, but it also seems to affect how low your speed can be to be considered "charging." It seems to take longer to couch a lance, for example, when riding a courser than a charger, despite the much greater acceleration.