Currently your character moves up and down on the ladder depending on where the player looks, so if you decide to look down during the climb, you'll start moving down the ladder even if you're holding W. I'd suggest to ignore player's camera during climbing and leave it simple: W makes you climb up, S makes you climb down.
