If you attack their caravan or village, it will lead to a war. Attacking one of their lords will also do probably.
It's likely exploit however, you should get massive negative relations with your liege as a minimum for doing so. If you should be allowed to do this in the first place.