I've just tried moving my 80-unit army between 2 castles a few times with around 8 inventory slots of food; 1st with 1 extra horse, 2nd with 2, 3rd with 3, and 4th with none. Each time my party travelled at a steady speed of 5.5. I'm quite sure that if I fill all inventory slots with Iron my speed will drop much more if I have fewer horses, but with a mostly empty inventory there appears to be no difference at all.
Pathfinding is the only skill I've found that affects map speed. My Deshavi has 6 in PF and when her health is in the red after a battle the party moves a lot slower. Morale helps map speed too. I remember once chasing an army around with exactly the same speed until we happened to pass through a village where I bought some honey and butter. The additional morale allowed me to catch them.
