This is how herd works. To have no penalty, You can have 1 animal + 1 riding horse per party member. The other animal can also be a horse, but it can't be 2 cows. You had 4 which is double the safe amount and just enough to get the maximum malus. 1 horse +1 cow, no penalty. 2 horse, no penalty. 2 cows, 50% penalty. 3 horses or 1 horse and 2 cows, etc, 50% penalty. 3 cows, 100% penalty. 4 anything, 100% penalty, even if 1 or all are horses. Each animal counts as 1 herd and only 1 horse can be a freebie, so at 4 horses, you get the 100% penalty of 3 animals or more per party member.