Check the morale of the individual soldier types that are leaving (can see this by simply clicking on them in the party menu). My guess is that you're warring with some faction, but have some of their types of soldiers in your party - this will give them very large penalties to morale and cause them to start deserting. You can see specific morale penalties/bonus' in the Reports screen.
Don't use Huscarls against Nords, Mamlukes against Sarranids, etc...If you go to war with a faction, you'll want to use mercenaries and/or troops from a different faction.