While the addition of the castellan building and the change to trade bound settlements have helped castles a good deal, there's still little purpose in owning one over a town. In my opinion, to make castles actually worth it to own, both castles and fief ownership in general needs several changes:
First, and most importantly, you shouldn't even be considered as a candidate for owning a town unless you have a lot of renown and influence within the faction. Being able the own a town on the first day we pledge allegiance to a faction as a tier 2 clan is immersion breaking (unless we had a significant amount of relations & influence beforehand) and it badly affects the progression of the game. Owning towns should of course be better than owning castles, but the fact towns are given out to any rando like candy makes it pointless to own a castle.
Secondly, castles are mostly defensive fortifications rather than a proper settlement. It's theoretical purpose is to provide security to the nearby domains of it's holder. Castles shouldn't even have loyalty and security since they can't even rebel (and therefore no fairgrounds and no loyalty boosting daily). But more importantly, castles should have unique buildings that provide bonuses to its villages. A village with a castle a building should make villages harder to raid (by having more militia and/or it taking longer to raid), another building should increase hearths or village production, and another should for example increase the size of the villager party, etc..
What do you think? What would you suggest to make castles better than they currently are right now?