I think one of the issues is that once a lord is beaten he has a very hard time to recover since (atleast from what I have seen) he starts out alone again with no people in his army. Making him easy target for the smallest bandit groups. My chat log is just being spammed with lords being captured by sea raiders/looters.
Not to mention, there are large groups of sea raiders plumbed up together, almost as strong as 300 men.