The issue is putting every form of progression and jamming it into the current skill/attributes/perks system.
They should've just separated combat progressions from your administrative or other personal skills; I don't even know why clan leader perk is there (we already have clan tiers? or the fact there's barely any perks applicable anyways). Governor perks could've acted more like a simple A/B perk path (maybe 5 levels or something) that you pick and develop the longer someone is a governor at X fief.
Or they should've treated the AI/companion skill path completely separate from the player as it still makes no sense why some skills only affect the clan leader, only the party leader, only the governor. Some perks will just never be of use even after managing to get a companion to get to that skill level.