Bandits
I think bandits should be always small in number, and never scale to match the player. But they are a major threat to the economy by hurting villages, and maybe caravans.
They should be small enough to be able to chase villagers. Currently they are large enough that most of the time, the villagers can outrun them.
So bandits will always be just a nuisance to a high level player, but never a real threat. But they will always be the bane of villages and villagers, and maybe even caravans.
Once you have the dynamic economy going, this is something for the player to do, to defend villagers and caravans from bandits. Or you can leave it to the scouts to take care of them.
Deserters
I feel that they should always be large and should always be a threat to the player. It depends on how you want to balance them.
They can be large, but you can always reduce the number of deserter groups on the map. If they are that large, the player should be able to outrun them most of the time. If they just travel from A to B and don't watch where they're going, they rightfully should be ambushed.
Or they can be average, but there are more groups of them.
So bandits hurt villagers and caravans, deserters hurt the player.