Well, there are more desireable buildings and less benefitting ones. There´s no need to build everything, so pick carefully. If you want a fullily upgraded town you´l have to wait quite some time. Rome wasn´t built in a day and so shan´t Pendor Local Village.
Especially since the gains the player gets from some of the buildings can add up dramatically plus it totally neglects the benefits of a dedicated engineer in your party.
You´re tinkering with the game balance. If you manually tweak and reduce building time you´ll automatically affect the AI as well which will generate more wealth, more money, better quality and quantity armies in both field armies and garrisons and hence raise difficulty as well.
Ponder in corkscrews about your first glimpse nonchalant innocent tweak. Usually there´s a halfton of unwelcomed sideffects showing up later on. The Face of Doom or Gollum knocking at your doorstep simply beeing a few of the minor inconvieniences which can happen.
Whatever your prefer. "That´s all I was tinkering with...!" won´t help once your precious savegame is screwed. So be careful. Make backups. Save often and in alternative saves. And don´t come crying if you get devoured by monsters. You called them upon yourself yourself.