Castles for now are just a stepping stone to get towns and owning one represents the start of the mid game. Later in the game the only use is due to the building that reduces upkeep of the garrison, so you can make them into strategic reserves to mobilize at need.
I understand people's suggestions to make them quality recruiting centers (everyone here has played Perisno and Pendor), but I think giving that role to the villages attached to the castle is a better solution, if anything because it gives you strategic depth: you can starve a faction out of its best troops by targeting specific villages, instead of having to launch a lengthy siege.
Best would be to extend the zone of control of the castles, and make them impassable to hostile forces. Either directly, by making any movement on them impossible (except sieges), or something more soft where the garrison acts like a village militia, helping Lords in range or intercepting you outright (potentially veeeery exploitable, you'd need the ai to intervene only when it calculates overwhelming odds), making it a true risk to just breeze through enemy castles and cutting you off from desperately needed reinforcements.