The -3 hit for owning a town of a different culture and the -1 for a governor of a different culture do present a challenge, but there are ways to offset them.
First, get a governor of the same culture. Either marry someone, or hook one of your brothers up with someone of the same culture. Check for companions of same culture. If you are lucky, you'll find an engineer who fits the bill.
Having no governor is better than having one of the wrong culture.
If your loyalty is in the crapper, cancel all production and set daily default to Festivals. This will start pushing your loyalty up.
Boost your garrison. This will increase security, which increases loyalty.
When you can build again, build a Fairgrounds to increase loyalty. A fully developed fairground will almost cancel out that -3. Other priority construction -- Orchards (or Gardens in Castle) and Granary, Workshops, Marketplace, Aquaduct. If food is not an issue, start with Workshops to increase Construction.
Handle every settlement issue and bound village issue to increase security. (Exception - Do not do the ones for gang members, as having these actually increase security.)
Stay close to defend your villages from raids. Raided villages produce no food, and no food produces unhappy citizens, which will eventually tank your loyalty.
And if you are in a kingdom, enact policies that increase loyalty. There are several, and these can counteract the -3 malus, too.
This isn't like Warband, where you could conquer and ignore. In this game, you have to actually be a ruler and take care of your subjects.
Eventually, you achiev a kind of equilibrium where you don't have to babysit them constantly inluess you are at war. Then you can move on to new conquests.