On the topic of making castles important, they really ought to have castellans who govern in the lord's place while he's out doing other things. A big part of this should be that they will grab a portion of the garrison and ride out in defense of villages when they get raided, which would prevent the very annoying situation of lords with tiny parties running around raiding everything with no opposition. They could also handle recruitment, training and feeding of the garrison, all subject to preferences laid out by the lord. Food should no longer magically remain at 30/100 days worth until a siege occurs but actually consist of stocks that are depleted according to the size of the garrison (and perhaps rationing policies, with garrison morale also being a thing) and replenished via the castellan sending out caravans with a guard drawn from the garrison. This would make location actually matter a whole lot more; trying to feed the garrison of a castle stuck deep in enemy territory would be a nightmare as the caravans are continually intercepted, resulting in a loss of part of the garrison and the funds provided for purchasing food every time. There could also be the option to harbour peasants when a village can't be defended, which would mean further depletion of food stores but quicker restoration of village prosperity when they return.
Managing a castle you own would mean sitting down with your castellan and sorting out budget constraints on food, recruitment and training, rules on how much of the garrison can be taken out for defending villages, ration policies, limits on harboring peasants, etc instead of just "hmm, do I want a prisoner tower?" Furthermore, there'd be extra strategic options like pulling a Theon; drawing out part of a castle's garrison with a raid and then besieging the castle while it's weakened. For this reason, many lords may elect to never defend their villages because they deem the safety of their personal household far more important than those filthy peasants.