I tend to end up with it on default for ages, then i suddenly remember the settings are there and change them up to 107 for a nice refreshing new challenge. At higher levels you might find it fun to run around building up your empire, in which case having low difficulty will make this much faster, and maybe more enjoyable.
However, at higher levels you still might like to have the danger of being knocked out fairly regularly, in which case a higher difficulty is necessary... it took a battle of me, 4 companions and 25 swadian knights in a siege against 250 khergits to realise that only losing a few of the knights and all the companions meant things must be too easy.