Update on this issue.
I've seen that the lords assigned to castle will stay as they shall.
Governors of cities will teleport all over the map after a few days.
Some of the bonuses they have will not apply, when they are not present, some will.
Like the "governor's culture bonus/penalty" will apply.
Bonus to prosperity and milita production will not apply - atleast not in my cases.
The security bonus that they have from skill will not be applied either.
While its not gamebreaking its rather annoying.
As you have to every other day check if the governor is at assigned spot or not.
The "fix" is to remove them from governor, choose no, then exit screen, assign them, exit screen and they are back applying various bonuses etc.
This can be done from the UI, and you dont have to travel all the way back to town.
Again I call it "fix" as you will have to check it once every other day, when they have teleported away.
I'm not 100% sure on this part I'm putting in, but it seems they are getting less steward skill than normal when they port, they still do get it, but much less(the city is +2 prosperity atm, and is fairly decent).
Governor "only" have 2 focus but still got plenty of pts to skill out yet.
And again when they are present they get an update to the skill every other day, and not once in a blue moon.
I havent tested with "noble governors"(like family) but just the companion governors, so I'm not sure if this is haveing an impact.
I tried to leave them in settlement manually, then assign the role, but they still ported.