I have the same problem. I noticed that if you lose a castle or a city that was under your command you lose relation with your clan leader. About 15 for castles and 40 for cities. And the thing is, you don't get any positive relation if you take them back. I don't know if this is intended, but it feels kinda bad when most of the time you are powerless to stop them.
edit: It seems you get some positive relation when the war ends, but not sure.