I'm having this issue in one of my games where I joined Vlandia. They instituted 2 laws, one being lawbearer and another I can't remember the name of, but essentially it states that any noble with low charisma loses 1 influence per day. Those were instituted within the first week of me joining the faction, didn't have enough influence to even vote one way or the other on them, now sitting at -30 or so influence and dropping with no way of reversal. Even with us being at war with Sturgia and me constantly fighting and beating their armies, I can't stave off the loss.