One castle comes with one village.I will try to balance out their cost for you.
The castle costs you 450 denars(250 if you have built blacksmith in it).Lets assume that you have a small garison in it that sums up to roughly 100 more denars per week.That makes your castle have a 550(350) denars upkeep.The village , even if its prosperous with a lot of people in it , isn't making such a huge profit to let you gain a high income.On top of that you will need to spend money to build things in those 2 locations.
By giving the castle away you get a happier (not to mention stronger) lord , you save the money that you would spend to improve it , you don't have to worry about garisoning it , and you are virtually not losing any profit since the castle with its village aren't contributing to it that much.