I think the issue with bandits is not found in the early game, but towards mid/end levels. Once the player gets beyong a certain threshold bandits and hide-out spawns overun the countryside.
Pseudo-historical accuracy should not take precedence over gameplay. Right now OP is rightfully complaining that he is losing control of his kingdom to an AI incapable of managing it. There's no dumber way of trashing your own game than by forcing players to trust bad AI.