It was the most common and most basic legitimate tactics of the steppe nomads for centuries - harassing with arrows, refusing close combat, retreating after depleting all arrows, replenishing arrows from carts and repeat the whole process. Time after time. Day after day. Non prepared enemy with less mobility and no proper ranged units was doomed.
Real life counter tactics i.e. during crusades was not to chase them with cavalry but to employ crossbowmen with big shields and shoot them with favorable kill ratio.
Restriction in Bannerlord should be you can't reengage again for i.e. 6 hours needed to replenish arrows, regroup and reorganise and let the horses and people rest. This time enemy army would be retreating to their castle decreasing number of times you could attack and harass them. Like IRL. And common sense.
But good finding of yours - reengaging constantly many times without any need of regrouping, resting, replenishing arrows is an exploit. A big one.
(It is actually stupid in TW serie when you can see you didn't loose your mounted archers, you see you break away from the opponent savely, you retreated your units savely and far from any enemy - and you see you have magically lost i.e. 125 warriors, just like that, killded by the wind.)
cheers