Windows does have optional language packs it doesn't install which can cause similar issues, though toggling the Windows language should trigger it to go fetch the needed packs. You could try it, but I doubt that's the problem (though with Windows you never know).
Have you tried running the game as admin? If Steam is installing to the default Program Files location you might be hitting permissions issues when the game tries to access the directory; right clicking on the executable, going into properties and checking the 'run as administrator' in the compatibility tab might help.