Castles provide a NEW elite troop in small supply to the owner or faction ruler.
Castles should straight up be the only source of "Elite" troops.
Doesn't make sense why Castle villages are special this way. It's also rather silly that literally anyone, from any Kingdom, can come by and pick up a Culture's Elite Troops from Castle Villages. That would be like a Germanic warband going into a Roman village and then suddenly walking away with a bunch of veteran Praetorians in their ranks. Only time you should see Elite Troops in enemy armies is when they turn coat, which should be rare.
Also Castles being the only source of Elite Troops would again make the Relationship mechanic meaningful, since you can't enter Castles when you have a negative Relationship with the Clan that holds it. My guess this is probably the most likely reason TW hasn't implemented it, since the A.I. Parties wouldn't be able to go to certain Castles, etc. Also they would need to re-balance Notables, so the current ratio of Elites to Normal isn't thrown out of wack.
They'll probably implement Castle Recruitment (despite claiming it's impossible) completely out-of-the-blue 2 years from now after some total conversion mod does it.
I think everything else about Castles is fine as is. They are good place to store large quantities of troops due to wage reductions. The problem is unlike a Towns, you can't hunker down and recruit units safely from within the walls. About the only other positive aspect of Castles is they don't rebel, but they effectively shut down at low loyalty, so is that really better?
I do find it annoying if you snipe a Castle your villagers won't trade with nearby Towns. Like really? You basically got two options for forming a Kingdom nowadays; snipe a bunch of fiefs as an independent Clan or get a bunch of Fiefs as a Vassal then break away. Also make sure you got lots of cash. I mean it's not impossible to defend a Castle/Town against a 1000+ Army with a couple hundred of your own, but the trial and error is not worth it. Also some Castle/Town scenes are more or less impossible to defend with small numbers, so yeah.