Actually your companions hang out in civilian attire while you're in town. Your main companion will follow you. You can talk to any of your companions and have them gather. Useful when fighting thugs and I'm assuming useful later on in development when you get ambushed in town or are doing quests in town where you have to fight.
You should be in civilian attire as well in towns and castles unless you're running the mod that allows you to be in armor in all cities and castles.
There's an alternative/non combat clothing option you'll see in your "I" inventory in the top center window. Open "I"nventory, look at the top center left is the helmet icon which shows your armor. Just to the right, top center, is a Robinhood looking cap. Click on the Robinhood looking cap and you'll see you and your companions "in town" attire options.