Personally I'd reverse the numbers of governors and owners.
Giving a owner penalty/bonus of 1 Loyality respectivly.
Also for Governor penalties, -3 for a wrong culture governor (AI clans usually not set governors too often in conquered towns at all) People are unhappy that their governor doesn't understand their culture obviously.
-1 for no governor at all. People want to have someone to represent them, but at least aren't angry about a wrong one.
+1 if the Governor matches the culture. Basicly nullyfieing the owner debuff. People feel represented well enogh. Who cares about the owner if the governor knows aboutt he peoples culture, somewhat keeps the owner "in check". Its probably not rare that a governor gives advices to the owner which things should be pushed through and which not. Obviously those advices not actually impleneted in the game, just a statement to things happening behind the scenes.
That way the AI would at least have a total of -2 in general for new fiefs. Thats a full loyality less and they should be somewhat stable. Also might help the game balance alltogether since.... for a proper war, the player literally has to join every single kingdom first, enable policies that help these kingdoms, then hope they wont abolish them until the player stabilized every single kingdom so these kingdoms can actually conquer anything. For the player kingdom that isn't a problem but... The player literally doesn't have to fear loosing fiefs, since if they are taken, raid a single village of that taken fief once its rebuilt and just wait for the rebellion and take it back. Thats getting even more of a problem if a AI kingdom has a brainstorm together and somehow enables a policy that decreases loyality... Then thats just effectivly "this kingdom can be ignored entirely" unless for a empire player vs a empire faction