Whie browsing the ideas in this forum, I came up with possible use of the castles, that would not harm current state of the game as much (I think).
Core idea
The castles, now useless, may become a way to reinforce your party, by having mercenary parties/sergeants/notables residing in them, just...