I have a similar issue, but on assault (haven't tried defense yet), doesn't matter if I join an attack or initiate one myself, usually crashes as I get closer to the castle, I managed to get inside a couple of times but it crashed at some point anyway.
The size of the siege doesn't matter, happened on 30 vs 20 sieges as well.
Happens on both 1.0.4 and 1.0.3 (haven't tried on earlier versions)