Agreed. It would be break the gameplay flow if you need to go back to your castle every time you want to make a decision though. I would suggest to add a minister and messenger mechanic. So if you go to your castle you can make decisions directly. If you are out in the field you can send a messenger (which takes more time and might be less efficient). It's just wishful thinking though, chances that it gets implemented are pretty low in my opinion because of the extra work.