The issue here really has nothing to do with difficulty. I always play at the maximum difficulty possible, and I typically wipe my save whenever I get defeated. If the game was simply too hard, I'd just turn it down. On the contrary, the issue is that the mechanic seems unfair and doesn't make much sense logically.
noosers, I understand what you're saying, but it doesn't really work out that way in practice. The AI isn't actually sophisticated enough to throw skirmishers at you. Almost every time the "join battle" mechanic occurs, it's when my party is chasing a smaller one, not vice versa. Furthermore, even if the enemy did throw skirmisher parties at me as a deliberate holding tactic, exactly how long do you think they'd survive? In real time, my battles are over in less than five minutes. Even if we're generous and assume that the battle takes longer than that, I'd be hard pressed to believe that some huge army covers several miles to attack me from the very beginning without even giving me a chance to retreat. This is especially true since my army is almost always much faster than whatever army joins my battles.
If I was given a time limit or something similar to defeat my enemies before the next wave arrives, thus simulating the distance they're covering, then I'd think it'd be a great feature. Unfortunately, I don't think the engine allows for that possibility, and, as it is, it's just frustrating.