Yeah, right now, it's pretty hard to position yourself such that you get to defend a siege battle - for understandable realism reasons. Unless you intentionally put yourself in a bad situation for fun, most likely, you aren't going to put yourself in a losing siege battle against impossible odds. At the same time, the AI is very unlikely to launch a siege assault unless they are fairly confident that they will win. (And, as you noted, even if your party is just the right strength to get the AI to attack while still having a chance at success, it's hard to get there just in time.)
This is not my original idea (thread on Remote Commanding), but, I think you should totally be able to command defensive battles when the enemy attacks your fiefs - even if your character isn't at the battle site on the campaign map.