Ok. I was a little incorrect on what exactly was happening. I did a few more test with cheats enables so I can teleport.
The start and end destinations are NOT, in fact, reversed. What's happening is the quest says a caravan is going to move from Town A to Town B after five of days. It does not, however start moving to and from those towns at any point. Instead, after the listed number of days, the quest will update saying the caravan is now moving from Town A to Town C. The caravan will spawn at Town A and indeed move to Town C. The events unfold as stated and the quest continues normally. Town B is never used in the quest.