I think the king also gets mad at you, loosing relation.
Somebody probably thinks it would be unbalanced for you to just lose your castle and then bid on yourself for a new one (cuz now no fief bonus), however I think you can not just give fiefs back anyways, so I would do that and try for a better situated fief, cuz factions probably won't help you.
I feels like TW really want you to just lop everyone heads of and never be a vassal or ruler.