This was the case in real life, too. However, in reality, they were not "levy soldiers" as with Khuzait horse archers, but rather people who had trained specifically for horse archery their entire lives, and were irreplaceable once lost. This is somewhat difficult to simulate in a video game.