Uh ... 20 pounds of styrofoam is a chunk of stuff about as big as a car ... and you won't be swimming with that either. Odds are it will hold you under water until you bubble, like being trapped under ice.
Speaking of which, wood shields worn on the back would be bad about holding your face in the water too. They would somewhat float, but it would rotate you so that you couldn't keep your head above water. The only thing in the water that's worse than being tied to something heavy is being trapped under something that floats. (Same way the worst thing in skydiving is to get tangled in your chute.)
Anyway, it's not anti-historical. Japanese naval warfare was known for putting heavily armored troops on ships, and that is well documented in art from various periods. Although the art is sketchy, it was seriously suggested that both Greek and Roman marines wore standard military armor. If they were afraid of falling in during some stunt-man maneuver, they tied themselves off with ropes (so they could pull themselves back aboard if they went in the water). The risk of drowning from wearing heavy armor was still much less than the risk of drowning because you couldn't swim with arrows in your body.