So I joined the Empire early in the game when they still had all their towns and castles. Then the D'Shar started invading our lands from the west and the Sarleons from the north leading to the Empire having only Cez left and me without a single fief, but only 20+/50+ relations with the lords...