Yes it can be tough, and unless you are some highish enough combat build, fighting 4 on 1 melee on foot can be quite daunting. I frequently turn down these quests unless I have a small enough party (I find that upto 40~50 gets you in most of the time) or until my archery has gone up enough points to be able to one hit headshot everyone when they drop their shields. For some reason even with 4 points in athletics I cant run fast enough in plate armor.
If you do get caught, and don't wanna risk a fight, try to run out by the exit and leave area. Then try infiltrating it again a few days later. Also, as instructed, I find that avoiding any enemy villages or lords on the way and trying to sneak in during the day seems to reduce the chance of having to fight.