I'm always skeptical of modern ideas around culture being applied to medieval style scenarios.
The game's time span barely covers a generation. In order for noticeable cultural change in the population to be affected, you'd need to have some sort of set of mechanics that touch on governance of exclusion based on culture.
Laws preventing militias and soldiers from being trained as per their traditions, laws ensuring new town notables being the governing culture and that the local culture are excluded. These kinds of situations would likely be ineffectual at instigating cultural change without the application of force, and for them to be believable, we'd also need an expansion of the rebellion mechanic - as nobody wants to see their culture subsumed by a foreign one.
Certainly it is possible for enforced cultural change to stick. For example, Normans largely assumed all governance roles in England for example for several centuries.
But as often as not the outcome of these kinds of situation is a hybridisation - where the new elites were themselves absorbed by the populace they governed (such as the Normans in England) to form a new cultural dynamic, or the new elites were absorbed entirely (such as the Franks in France, Lombards in Italy or Goths in Spain). This is why up until the emergence of nationalism in Eurasia... conquest of populations didn't often result in conquest of culture in the way we see it described today. There were still Greeks in Anatolia, Still Christians in Egypt, still Latins in Andalusia, still Welsh in Wales... many centuries after their conquest and exclusion by outsiders.
That, and of course, in feudal societies, culture itself often took on a secondary role. The Scottish nobility owned lands and titles in England, the English nobility owned lands and titles in France. German, Dutch, Scandinavian, French, Spanish nobles and royals often owned lands in each other's 'technical' realms. A serf doesn't always know, let alone understand the language of the person who owns them.
More likely in a feudal system like Calradia, it is more effective as the owner of a fief, to utilise the local feudal bonds to call local militias, local petty nobles to govern through oaths of allegiance.
I think the suggested modifier would logically result in a reduction of loyalty, because the locals are being excluded from their own governance. I think the loyalty penalty should be tied to prosperity, not culture. People are usually happy when they're wealthy, no matter who owns them.