I think Swadia's current predicament in your game is based simply on bad luck and the dynamic turn out of events in your particular game. On my last game, Swadia did fine and managed to hold all of its fiefs for most of the game, which lasted 360 days. In my current game, Swadia was down to Dhirim, Suno, and some castles, because they were attacked from three different kingdoms in a very short time frame. They would have gotten back on their feet, but when my clan declared war on them and made me marshal, it sealed their doom. My previous game was nothing like this, in that game Swadia seldom lost a fief and was usually only at war with the Nords and Rhodocks.
Although I'm don't know for certain, I don't think Swadia's(or any factions) lords are predisposed to being divisive or wimpy, I think its just how things turned out over the course of your game. As stated already, Swadia has fiefs that border all the other factions(except Sarrinids), which makes it easy for them to get attacked from any angle.