You really shouldn't be getting any weapon or any experience (maybe roguery?) for chasing after and slaughtering the retreating soldiers who never fight back at all. And the group that retreats should disappear from the map, as they melt way into the hills, so to speak. So there's no regrouping of 1-2 dudes who suddenly feel brave enough ten minutes later to face your fully reorganized army?
Nah.
Nah.