I agree this is an issue.
I focused on trade instead of joining a kingdom and once I reached clan tier 4 I decided that me and my clans men/women can take at least one castle since our combined parties cost 1300 per day. Not the case, only kingdoms can work together.
If the developers want to keep the 'army' feature to kingdoms, then at least allow party leaders to be given instructions like attack this place or something.