Ah -- I should have explained that. The red score is the team score from killing enemy agents, the yellow score is the score from other objectives -- caravans crossing the map, or possession of cattle over time. (The X/Y score in livestock raid represents the current possession of cattle, but may not be working right now). They determine who wins the match and who gains/loses a conquest point on the map, but don't have a direct impact on your character.