As far as I've noticed, attacking a lord of a kingdom without a war going on, or raiding a village in a similar circumstance nets me negative influence. Seems to be a penalty for warmongering or something functionally similar.
Also, sometimes Proposal buttons (propose peace/war, propose a policy) are clickable when you don't have enough influence (happens when you open the respective screen of the kingdom menu, the first entry that gets automatically opened becomes clickable), so you might've accidentally given one of those a click.