I am fairly certain that the reason for this issue is that they removed simple XP gain for leveling and instead bound it to the system that weapon proficiencies in Warband had.
When you fight an actual battle (that is simulated) there is no collective XP for the win distributed among all troops after the battle like it was in Warband. Troops have to hit with their weapon in order to gain some XP and it is always the same amount, i.e. never a bigger amount for killing an elite enemy. In many battles not nearly all your troops see any action.
In "simulated battles" however (battles that aren't simulated), there's some function used to calculate XP for leveling that is arbitrarily given to your troops (so more similar to Warband).
I think this is the reason, but of course I don't know the code. They could adjust the XP given to more accurately reflect what you usually get on average when you fight the battles even though it's impossible to predict how much each individual troop would have hit with their weapons (or how much they would have run around or whatever is needed to level Athletics or Riding).
Perhaps I'm completely wrong. Perhaps generic troops do get XP for kills unlike heroes, who knows. I think the whole XP and leveling system is a bit of a mess.
You're completely wrong.
Simulated battles don't give XP arbitrarily to your troops. Simulated battles simulate hits from one side against targets on the other side. You gain EXP per hit similarly to normal combat. The difference is that it doesn't actually know anything about your weapon or armor, it uses your level instead for the damage calculation, it doesn't give a shot difficulty bonus or a speed bonus, and it gives 8 times as much experience as if you had hit for that much damage against a player in regular combat.
Troops get XP for hits, and bonus XP if it's a killing blow, in the same fashion that heroes get weapon xp.
Well, maybe you're not completely wrong, the whole XP and leveling system is a bit of a mess.
There are some things that people can know about autoresolved battles. One thing is that it tends to start from the top of your roster and work its way down. If you want characters to get experience, put them at the top, if you want characters to have a first strike, put them at the top. This mostly matters when sides have a similar number of troops, or if you as an attacker outnumber the enemy. If you have all of your top tier troops at the top of the army and your recruits at the bottom, you will probably suffer few losses against a smaller bandit force, but your recruits will get less XP. This order also matters for the defender.
I tend to like to keep my party ordered in descending power, but when I'm doing map battles I move my recruits nearer the top, and when I'm doing hideouts I move them to the bottom. When a battle isn't decided on the first strikes everyone tends to get a reasonable amount of experience.
I basically autoresolve 90% of my battles, I play them out only if I'm liable to lose by autoresolving. You tend to lose more troops with autoresolving. In a real battle a low level horse archer will kill a high level spearman just because the horse archer can stay at range and shoot the spearman when he can't catch up or reach him. In an autoresolved battle, the high level spearman will kill the horse archer because they will just essentially swing at eachother and the high level troop will deal high damage to the low level horse archer, and the low level horse archer will deal low damage to the spearman. It won't care about relative speed, range, shields, whatever. Just level.
I actually don't mind it so much. It would be better if the model was closer to gameplay results. In reality you kind of play 2 games, one is the simpler map mode autoresolve, the other is the more complex battlefield game. If I'm playing caravan games I'm more interested in trade prices than manually fighting bandits, especially when my hero has no combat skills.
The model can be improved, EXP has a lot more problems than just the autoresolve experience. Like lots of things in the game, the fundamentals are OK, but the numbers and formulas need to be tweaked. The fact that you need 10 million experience to reach the last perk level when you get it hundreds at a time for instance is an issue. Or that you need 1.7 million experience to reach the same skill level as your troops. Similarly, how your player can't get experience from autoresolution but your troops level super fast.
Like in my current game I'm playing a trader and the only time I'm playing out battles is in hideouts, which means that I'm basically leveling up only shielded infantry because so many hideouts have javelins and bows. This doesn't matter in the open world though because archers and cavalry have no particular advantage when you're autoresolving. Similarly, I never advance units to cavalry because this consumes a horse that I don't get back when a unit dies. However, if I have infantry with a horse in my inventory for movement on the map, if my infantry dies I don't lose the horse. Again, it doesn't matter in the actual battle whether I have cavalry or infantry, if their level is the same, they'll be equally effective.