Yes, currently all routed parties become part of Swadia and if they run into any other rouge faction (sea raiders, mountain bandits, looters, forest bandits, tundra bandits) they will fight. Of course if you intervene you are technically helping out the Swadians, and like it was said before they always head to the nearest Swadian castle/town. This needs to be fixed, I think it was just an easy mechanical set up like it was mentioned above but it has significant and unwanted side effects so it's something that needs to be addressed.