Loyalty doesn't really matter for castles since they can't rebel. Prosperity is marginal and usually not worth fussing about.
Governors can be a big deal for towns though, especially if they have the correct culture to offset any culture penalty. They also habitually raise relations with the notables over time, even faster with one perk (Meaningful Favors).