I kind of feel the same way as the OP. I kind of find there are certain areas of the map I like to play in and if I am given a castle that is too inconvenient, I tend to let the castle just wither on the vine and hope for a better one later. I just don't feel like wasting the time and effort going back and forward to a castle half way across the map.
Also, castles are neat and all but at the end of the day, I want to rule a city. However, if you have other fiefs when a city comes up, I think you have a much lower chance of even becoming a candidate up for vote much less getting the city.
Honestly this is why I kind of miss the "keep what you capture" mechanic. At least with that mechanic, you had a way to choose where you wanted to set up shop.