You don't have to fight the Swadians.
Just renounce your oath first. Then build up your own empire.
Only when you're strong enough attack the Swadians, and keep their lords as prisoners.
That way the get less and less mobile armies. It's a good way to defeat them.
But don't keep the nobles in your party. Drop them off at a castle (preferably with prisoner towers)...