Also, I tried starting a war by raiding a village, which works, but my influence is killed! -300 influence! I had like 17 influence, went down to -250 something.
If the main story of the game is to get rid of empire settlements, there should definitely be quest dialog to declare war without ruining influence.