The attacking teams archers are actually in a superior position at the start.
No castle would ever be built in a location where attackers can swarm up the nearby hills and easily fire into it....The fact that the ruined gatehouse is the largest and tallest bit of the castle is madness.
A defending archer should never be in a position where they are getting shot from the sides and above or from in front and above by the enemy archers prior to the enemy capturing the walls. Thats the entire point of walls, they were designed to protect defenders from this. If someone did build a fort by a hill they would have built a wall high enough to prevent archers on the hill shooting in so easily. Instead this map encourages this to happen.
A few invisible walls would be an extremely quick fix for the devs without actually altering the appearance of the map. Prevent the attackers from climbing the ruined gatehouse or flanking up the sides on the hills. This would stop them firing along the walls and down into them so easily.