Yeah after some time, if you have too many units in your party they will leave, so make sure not to pass your limit of troops and to have some food for them ready. They also leave because of low moral.
And I don't know about companions. I think they leave if you do things they dislike, but they won't leave because your party is too large. You also can't have to many of them since there is a limit.