if you follow the story at that point you have about a thousand warriors travelling in the sea. About half are with Sven, half with your side.
Then your side decides to do a *secret meeting* with only the captains. And you get ambushed. So it would be like 500 warriors attacking less than 50 on your side.
Yeah better run and fast
TLDR: run