To be honest, there should not be an option to do so. You're a vassal, your direct ruler may appoint a castle to you both as a gift and as a punishment. Or maybe he thinks nobody but you can handle holding said castle against enemy sieges. If you don't approve your rulers decisions you can break your sacred vow of loyalty and leave the kingdom all together.
But what I do approve however is for AI to stop taking castles far into the enemy territory despite the lack of garrison there and focus on adjacent towns instead.