You're sort of right
I'd say the issue is more how quickly Villages recover. Honestly I think villages should have to be manually "rebuilt", that way you have to decide more whether you want to stay offensive or defensive. Do you keep attacking your enemy with what you got or do you go back and repair your villages so you can get recruits later? Basically until a Lord rebuilds a village, it's going to remain a burnt crater or at least it should take an extremely long time to recover on it's own (like a year).
This could be something you, the player, do during peace time too. Since there's nothing to do really besides trade and arrange marriages, if that.
Also if some small band of mercenaries does eff up your villages and you get there a little too late, it's not a total waste to show up there like it is now. So annoying to just watch your villages smoke as it were for a couple days and not do anything about it, especially since you're encouraged to recruit from your villages.
Rebuilding villages could be good way to add some much needed way of gaining XP for Medicine and Engineering too.
I do recall being in a pretty long war with the Aserai once, where I got really fed up with my fellow vassals incompetence. So instead of trading fiefs back and forth with armies, I just took my well equipped party and went about raiding just about every single Aserai village. Needless to say the war dramatically shifted in my Kingdom's favor since the Aserai really weren't able to field many units after that.
So raiding is somewhat effective thing to do in-game, especially since defeating single Parties often does more good in the long run anyways. Also really hurts foreign invaders since it creates both Loyalty/Security debuffs to Towns I believe.
My biggest gripe with raiding is how little loot you get. In Warband raiding is very profitable, so yeah it's a bit heinous, but think of all the loot! In Bannerlord raiding just makes me feel like a petty ****. There's no real benefit to the player other then a kind of middle-finger to the would be owner.
Be nice if Bandits and such raided villages, rather than simply harass travelers.
Also be neat if you could "Sack" Town/Castles, i.e. instead of taking over a Town/Castle you run off with a ton of loot/gold and leave it a smoking ruin for your enemies to rebuild. Seems really odd to devastate a Town/Castle and then take it over when everyone should hate your guts for doing so. Then you could apply my aforementioned rebuilding mechanic to them as well (again more ways to get much needed Medicine/Engineering XP) and do something besides murdering all the common folk of Calradia all the time.
Heck you could actually do a true "good guy" Saint playthrough where you just go around rebuilding all the villages ruined by war. Make future Jeremus proud.