Yeah. This is irritating as hell. I've resorted to not fighting near villages at all. I just let them go and then chase them down again. As for Hideouts, I have to remember to reassign the units back to default before going in.
needless to say, I forget sometimes and half of them get wiped out.