I also tagged them as a special devision. Currently I can field over 70 Huskarla and Hesir. And of course I don't use them in the first line. I lead them myself, mostly in flanking maneuvers to crush the enemy shield wall or to breach a wall. Only for those special purpose they are deployed. Usually I don't loose more than one or two of them in a battle. So that's not my Problem.
What I meant was loosing what you called mid tier infantry. For me those are Rekkr and Thegn. The issue about them is, that they are much more effectiv then Hirthmatr and Dengri and thus levelling up much faster then those. Since my army is an pyramidal system with e.g. 100 Hirthmatr, 50 Rekkr and 35 Huskarla I have a constant growth in the ranks of Huskarla (wich is good) but a decline in the ranks of the Rekkr. This is because of the fact I mentioned before: The Rekkr and Hirthmatr are in the main line, but the Rekkr are leveling faster because they get much more kills. So my elite is stable or even growing, but my mid tier declines while campaign goes on. That's what grinds my gears, since I have to stock those up again after the end of the campaign. And to level up some Rekkr costs much more time then to level up some Hirthmatr.
But I can truly understand that you hate to loose those skirmishers. I loved to use them in Brytenwalda. But also there they already have been hard to replace, especially since the skirmish line is a dangerous place and thus they naturally suffer heavy losses even if you miscalculate their position between enemy line and your own main line just a tiny bit. So in VC I didn't bother yet to deploy skirmishers. Most of the units at least have Throwing spears so I usually just command them to throw them in the right moment thus they use their throwing weapons to some good effect. If I would let them "fire at will" they would just throw them much to early and so waste them. The AI is quite stupid there.