I just noticed that Harmony was listed as a dependency which I'm not currently using. I removed references from the SubModule.xml but still the same problem.
I've confirmed that the mod is working and can use it whether debugging or just deploying and then running the game without attaching VS. I just can't save or load games when attached to VS.