relation can increase/decrease depending on several factors and events, some connected to the storyline and others to sandbox.
check your message log (in-game), it may have some hints.
also remember the game tends to balance it out to a neutral position (so relations tend to zero over time), unless you do stuff to push them to positive/negative.
if you feel something is not working properly you can visit the support section and create a thread there. Include your rgl_log.txt, savegame, description of what is happening, info on your computer, if you use any tweaks/cheats/etc