If you fail the quest follower stays in your party he is count as hero and wears your colors. You can talk to him and tell him that you don't longer need him, however he doesn't go away and just sits in your party.
The only way to get rid of him is to leave in a village.
Not sure if it's...