I believe I left because he was defeated and captured, however the quest "Enlisted in a lords army" still remained. There was no dialogue option to leave the lords army when I later returned to talk to him.
I could of probably enlisted in another lords army and then left to fix the problem but didnt want to loose my massive army. Instead I fixed the problem by using the cheatmenu to change my banner, which made the banner show up on the world map.