Yes the spam of small armies raiding your stuff is quite annoying. IN THEORY it should slow down after a bit because they will run low on recruits, but really all that happens is they show up with armies full of bad units. But you don't need good units to raid villages. So the annoyance continues infinitely.
This pushes players into two behaviors: mass execution (which I don't like), and building cheeseball archer armies that don't take casualties. Mass archer armies are utterly devastating vs the low tier infantry armies the AI lords bring at you, but from a tactical standpoint they can get rather boring (horse archers are fun if you take on really huge odds though).
But on the other side, if you nerf lord recovery then that makes snowballing more of a problem. So maybe, instead of making it harder for lords to recover, make their behavior more defensive if it's clear their kingdom is on the back foot. Slows down snowballing and makes the game less of a headache for players.
RE bandit camps, key is good armor and bringing good archers and being a good archer. Then dodging enemy projectiles like a pro while you kill them at range. I don't know how the hell you would do those camps as a melee group tbh.