I have to add my support for this, or, rather, the scaling difficulty which doesn't do what it's supposed to do. Basically, at low levels these are hard, but far from impossible fights, since you only get about two spawns per spawnpoint, meaning you run back and forth through the whole 'level' about twice, killing maybe 20 odd bandits. Which is totally cool with good companions and proper use of tactics.
The problem comes when you're level 20+ - then the hordes of bandits literally becomes an avalanche. 50-60 bandits is NOT fun to kill with just yourself and five other guys, even if those five are high-tiered soldiers. Bandits are also exceedingly well equipped, generally having a ranged/throwing weapon, shield and melee weapon. Their only weakness really is their usually poor armor. Steppe bandits, for instance, are absolutely impossible to kill, with their accurate bows, shields and clubs (man I hate maces and the like - one mistake and you're knocked to the ground and pummelled to death). The only lair I can destroy with some degree of safety is the Mountain Bandit lair, since they all spawn within a single village, which can be monitored from the nearby hillock. This is actually a fairly good fight, one I don't mind. The other areas, however, are murder.
How to fix this? Well. One thing would be to implement what has been suggested already, which is to say limit the respawning and instead make the run a once-through. So if there are 60 bandits, and 5 spawn points, make each of the spawn points crush out 10-12 bandits when approached (they pour out of a house/a cave/etc). These are then killed, and once the last one is dead, that spawn is finished. And so the player moves on. Not only will this make the whole ordeal faster, it also seems a lot more intuitive and, ultimately, a lot less frustrating (nothing is more annoying than running back and forth killing a few at a time, only to be killed by the last bastard after like 20 minutes of fighting).
Another alternative might be to make your skills matter. Say, for instance, 1 more man for every point of Tactics or Leadership? Or, since they're in such a hard-to-get-place, how about one more man for every point in Path Finding (party skill, of course)? Since skills also increase with levels, the progression should be fairly natural (+3 men in the beginning, +5 later, up to +7 in very late game, with extra bonuses of course for personal skill and so on).
Finally, simply make the amount of men taken correspond directly to the amount of enemies. Level 5 with 20 enemies? Sure, five men is enough. Level 25 with 56 enemies? Well, maybe 10-12 men should be enough. I mean, think about it. It's a narrow defile, fine - this is fine if the total amount of bandits is very low. But when we're getting up to armies of half a hundred, I'm really wondering how they're making due with the tiny little entranceway (what with their carts and horses and mules filled with loot!). One'd think it's be enough to slip through a couple of one's own men as well...
Anyway, yeah. While we're complaining: sneaking into town and getting discovered and then surrounded after half a second and losing, annoying isn't it?