Did you try to make peace with them? Because of their low menpower they are constantly fleeing from any of your parties and most likely eaven from your villagers, but they have nowhere to go and so they are stuck. So, making peace with them should help them to spread out and leave that area.
I observed similar behaviour by beaten factions (i eaven saw a bunch of lords and mercs with 0 men, all cuddled together on a peninsula) without settlements and sooner or later all lords join other factions until only the ruler clan is left.
The northern empire will eventually join entirely the strongest remaining part of the empire when the mainquestline goes far enough. In my savegame Lucon and Garios are meanwhile vassals of Rhagaea.