Your relationship will slowly (and seemingly randomly) recover if you own the fief. It does not replace the notables.
This might be because my character has high charm, or it might be the high security bonus to relations. The latter pops up in the event log but I don't get the top banner with any notables' relations improving, whereas I sometimes get a bunch of "Your relationship with X of FIEF has improved by 2 to 52" if I'm in the area or resting in a settlement nearby. It might be one of the parties in my clan saving villagers from bandits (sometimes the improvement is like +12 relations, which is damned big) but I can't be sure.
But at any rate, if you own the fief, yes, your relations will slowly improve on their own. Even if they utterly despise you.
I'm actually fine with this, because raiding has never not been annoying and imbalanced (in one way or another) in every Mount and Blade title, spin-off and mod. Remember getting your first village in Warband and having to camp on it to stop it from being raided from across the map? Remember your army peeling off parties to raid some village rated as "Poor" just so they could make it "Very Poor"? Well, that was Warband, and let's put the past behind us: what about raiding a village with a silver mine in Bannerlord, only to discover that SURPRISE there are only two bits of silver ore but eleven bags of grain and some cheese as a result of your larceny and arson? This is spite of there having been fifteen silver ore on the village market to purchase. It is absolutely no exaggeration to say that you lose potential profit by raiding instead of purchasing.
The entire system needs to be reworked because it is nothing more than a trap. Even if you don't give a damn about the village, you're better off engaging in some other activity to gain income because the actual payouts from raiding are objectively terrible. Worst is that raiding is a fixed time, mostly static amount acitivity that does not scale with anything except (maybe) your Roguery skill and some of the perks. Beating up groups of looters pays similar amounts, even before factoring in the risks involved and the opportunity costs of lost relations.
I'm thinking they really just want to leave it as a noob trap though.
Raiding is not meant to be a primary way of making large amounts of money. It's meant to be a form of economic warfare. If you kill the villages, the town/castle relying on it suffers in the form of reduced food (leading to reduced prosperity, garrison, and militia) and reduced goods (leading to further reduced prosperity). Reduced prosperity has a direct effect on the amount of money a castle/town provides to the owning lord.





