As much as my initial reaction is "git gud" I see where you're coming from. I definitely think that if only horse archers remain in an enemy army, they should retreat. I've seen so many battles where there are 3 horse archers left just running around knocking people over for 3 minutes before they die. It's ridiculous.
Even judging the game by historical standards, horse archers historically weren't the ridiculous killing machines you see in games, they were a specialist part of some armies that was mostly used in the scouting and skirmish phase of a battle which games almost never depict. If an army of the world's best horse archers arrived in a battlefield as tiny as the warband ones and faced an army of shielded infantry, they would probably be forced to retreat. Even completely mounted armies like the Mongols, the Scythians and even the Oyo in West Africa had to coordinate horse archers with heavily armoured lancers, and they could fire literally millions of arrows over the course of the battle, refilling their quivers from a baggage train, and skirmish with an army for months.
Horse archers were a gamechanger, and every region of the world that had to deal with mounted nomads has been in turmoil since the 1100s, but the reason they were so dangerous is something you can't depict in a game like bannerlord.