B Info Shader Shader Stuff- HLSL instruction limits

[Swyter]

Here's the translated message with a working UV sprite atlas shader. Just for you and those interested coming from search engines.

You may want to do it your way by using your own shaders. Here are two ways I can think of right now:

1. Use every available texture slots in the material, so you'd be limited by the existing 6 slots.
The good thing is that you already have the code above, the one I wrote in my other post. It's about alternating between samplers, that in the pixel shader aren't more than numeric values from 0 to 5.

2. Paste all the frames into a single texture, vertically, like in a film strip, and just add a displacement in the pixel shader using

time_var

as modulator. It may take a while to thought it out, but the freedom that this method gives you is inmense.

The second method looks easier to me, and as you've said before, more feasible. Many games make use of that way of animating textures (that I know of). In fact I made my dynamic main menu just like that, with a displacement of the texture by the X axis.

So to get a working texture animation should I do something like this?

float speed = 1.05; // displacement speed
tc.x -= speed*time_var;

Thanks in advance.

/Mark7

In this example I'm using the vertical axis, but it doesn't really matter, just be cautious when changing coordinates...

[color=navy]/* sawtooth staircase pixel shader -- created by swyter */
/* online GLSL live proof-of-concept here: http://glsl.heroku.com/e#11374.0 */

  //          0.f
  //       _  A
  //      |_| |
  //      |_| |
  //      |_| |
  //      |_| |
  //      |_| | Y
  //      |_| |
  //      |_| |
  //      |_| |
  //          V
  //  0.f <-> 1.f
  //       X

  
Vertex shader:  
--------------[/color]
  
  int frame_count  = 8;                          [color=navy]// total number of animation frames[/color]
  int frame_height = 64;                         [color=navy]// per-frame vertical pixels, fixed[/color]
  int total_height = frame_count * frame_height; [color=navy]// vertical pixels of the whole image[/color]
  
  float sawtooth = time_var - floor(time_var);   [color=navy]// goes from zero to one and starts again, it subtracts itself (57.3 - 57 = .3)[/color]

  uv.y /= frame_count;                           [color=navy]// scale the coordinates reducing them to show just the first frame (1/8)[/color]
  
  uv.y += floor(sawtooth * frame_count)          [color=navy]// expand the count from 0.0-1.0 to the total number of frames 0.0-8.0, while getting rid of the decimals, leaving an integer value like 3.7 -> 3.0[/color]
  uv.y /= frame_count                            [color=navy]// scale it back normalizing it to the 0-1 range, but now it will be jumping in discrete steps without transitions or sliding[/color]

  
[color=navy]Pixel shader:  
-------------[/color]
  
  tex2d(diffuse, uv);                            [color=navy]// do the sampling in the correct coordinate, as always, there's no need of changing anything at all[/color]

I've created a tiny workbench so you can play around directly with the parameters in your browser, beware, this is GLSL, there are some variations respect to HLSL. It's done with colors instead of UV coordinates, and plays a frame per second, should be helpful:

http://glsl.heroku.com/e#11374.0

 

[ShaunRemo]

Hello, I'm not sure if this is the place to ask for help but I was wondering about the skybox_shaders ability to be rendered infinity (no clipping)
I used the shader to create large scene borders.

Spoiler

This is from one corner of a large map looking across to the other side
However the border no longer processes fog relative to the player and instead the map origin and therefore looks slightly out of place.

Here you can see just how mismatched the fog levels are from the border to the scene.
please note the far edge of the scene is being clipped whilst the scene border is not!

However in order to get it working  the Render 1st flag and No depth Test flags are used, I'm not sure if the flags are to blame or the shader but as you can see the fog displays incorrectly and it has render issues such as rendering through itself or allowing objects to render over it.

So I guess my question is, is it possible to create a new shader that is able to render as far as it can (no clipping) without the adverse effects of the skybox_shader?

Any help would be greatly appreciated as I endeavoured to fix this myself yet have no idea about shaders or how to write them.

-ShaunRemo

 

[Swyter]

However in order to get it working  the Render 1st flag and No depth Test flags are used, I'm not sure if the flags are to blame or the shader but as you can see the fog displays incorrectly and it has render issues such as rendering through itself or allowing objects to render over it.

So I guess my question is, is it possible to create a new shader that is able to render as far as it can (no clipping) without the adverse effects of the skybox_shader?

Any help would be greatly appreciated as I endeavoured to fix this myself yet have no idea about shaders or how to write them.

-ShaunRemo

The depth buffering is an essential part of the hardware fog.
Try removing No depth test and using the render order to render it first instead.

You can't avoid clipping, that's all configured by the engine. But there's a special flag for mesh objects called At camera origin, that is used by all the skybox dome meshes, which moves the mesh with the player. Otherwise the skybox meshes would clip against the frustrum.

The best solution for me is just trying to make the outer impostor skirt a bit smaller, the player won't notice.

 

[ElPadrino]

Oh I searched that 

 

[ShaunRemo]

The depth buffering is an essential part of the hardware fog.
Try removing No depth test and using the render order to render it first instead.

You can't avoid clipping, that's all configured by the engine. But there's a special flag for mesh objects called At camera origin, that is used by all the skybox dome meshes, which moves the mesh with the player. Otherwise the skybox meshes would clip against the frustrum.

The best solution for me is just trying to make the outer impostor skirt a bit smaller, the player won't notice.

Yeah I was just excited to have the potential of the view distance and though there might be a solution.

Spoiler

This is with No Depth Test turned off, it has artefacts + the original issues 

I'm not sure if I worded it wrong but the skybox_shader does have the ability to not be clipped and I wanted to know if I can extract that and make a new shader that doesn't render incorrectly.

Spoiler

As you can see the far corner of the scene has reached the view distance and is being clipped but the the border isn't.

 

[La Grandmaster]

Ive been messing around with shaders recently and have managed to create a couple waving flag ones. I have been trying to get Barabas' parallax one working,  but I constantly get an error, the error says invalid subscript 'w' 

I believe it is because of the .w at the end of the below line

In.Tex0 -= fHeightMapScale *heightmap * In.ViewDir.xy*In.WorldNormal.w;

any suggestions on how to fix this? I have removed it, which fixes the error but this will obviously have effects for the actual shader. I am pretty clueless about HLSL but learning slowly, does the w stands for the texture coordinate, as in u+v

Any help would be massively appreciated 

Barabas' full shader below

PS_OUTPUT ps_main_bump( VS_OUTPUT_BUMP In, uniform const int PcfMode )
{ 
	PS_OUTPUT Output;
	
	
	float3 normal;
	
	//temp should make new shadere definitions
	bool use_parallax = true, use_parallax_shading = true;
	Output.RGBColor.rgb =1.0f;
	float shade;
	
	if(use_parallax)
	{
		float height = (tex2D(NormalTextureSampler, In.Tex0).r);
		float fHeightMapScale = 0.025;

		height = (tex2D(NormalTextureSampler, In.Tex0).r);
		float heightmap =  .5f-height;
		
		In.Tex0 -= fHeightMapScale *heightmap * In.ViewDir.xy*In.WorldNormal.w;
		
		normal.xy = (2.0f * tex2D(NormalTextureSampler, In.Tex0).ag - 1.0f);
		
		normal.z = sqrt(1.0f - dot(normal.xy, normal.xy));
		shade = saturate(dot(In.SunLightDir.xyz, normal.xyz));
		
		if (shade > 0 && use_parallax_shading)
		{
			height = (tex2D(NormalTextureSampler, In.Tex0).r);
			heightmap =  1.0f-height;		
			float startoffset = fHeightMapScale*In.SunLightDir.xy/max(0.2,In.SunLightDir.z);
			int numsteps=20;
			float step = heightmap*float(1.0/numsteps);
			float stepheight = 1.0f+.5*step;
			[unroll(numsteps)] while ( stepheight > height)
			{
				height = tex2D(NormalTextureSampler, In.Tex0+startoffset*(heightmap-step)).r;
				stepheight -= step;
			}
			if (saturate(stepheight)<height)
				Output.RGBColor.rgb *=0.5*(height-heightmap)+0.5;
		}
	
	}
	else
	{
	normal.xy = (2.0f * tex2D(NormalTextureSampler, In.Tex0).ag - 1.0f);
	
	normal.z = sqrt(1.0f - dot(normal.xy, normal.xy));
	shade = saturate(dot(In.SunLightDir.xyz, normal.xyz));
	}
	
	float4 total_light = vAmbientColor;//In.LightAmbient;
	if (PcfMode != PCF_NONE)
	{
		float sun_amount = 0.03f + GetSunAmount(PcfMode, In.ShadowTexCoord, In.ShadowTexelPos);
		total_light += shade * sun_amount * vSunColor;
	}
	else
	{
		total_light += shade  * vSunColor;
	}
	total_light += saturate(dot(In.SkyLightDir.xyz, normal.xyz)) * vSkyLightColor;
	
	#ifndef USE_LIGHTING_PASS
		total_light += saturate(dot(In.PointLightDir.xyz, normal.xyz)) * vPointLightColor;
	#endif

	Output.RGBColor.rgb *= total_light.rgb;
	Output.RGBColor.a = 1.0f;
	Output.RGBColor *= vMaterialColor;
	
	float4 tex_col = tex2D(MeshTextureSampler, In.Tex0);
	INPUT_TEX_GAMMA(tex_col.rgb);

	Output.RGBColor *= tex_col;
	Output.RGBColor *= In.VertexColor;

	OUTPUT_GAMMA(Output.RGBColor.rgb);

	return Output;
}

 

[Swyter]

Ive been messing around with shaders recently and have managed to create a couple waving flag ones. I have been trying to get Barabas' parallax one working,  but I constantly get an error, the error says invalid subscript 'w' 

I believe it is because of the .w at the end of the below line

In.Tex0 -= fHeightMapScale *heightmap * In.ViewDir.xy*In.WorldNormal.w;

any suggestions on how to fix this? I have removed it, which fixes the error but this will obviously have effects for the actual shader. I am pretty clueless about HLSL but learning slowly, does the w stands for the texture coordinate, as in u+v

Any help would be massively appreciated 

Barabas' full shader below

PS_OUTPUT ps_main_bump( VS_OUTPUT_BUMP In, uniform const int PcfMode )
{ 
	PS_OUTPUT Output;
	
	
	float3 normal;
	
	//temp should make new shadere definitions
	bool use_parallax = true, use_parallax_shading = true;
	Output.RGBColor.rgb =1.0f;
	float shade;
	
	if(use_parallax)
	{
		float height = (tex2D(NormalTextureSampler, In.Tex0).r);
		float fHeightMapScale = 0.025;

		height = (tex2D(NormalTextureSampler, In.Tex0).r);
		float heightmap =  .5f-height;
		
		In.Tex0 -= fHeightMapScale *heightmap * In.ViewDir.xy*In.WorldNormal.w;
		
		normal.xy = (2.0f * tex2D(NormalTextureSampler, In.Tex0).ag - 1.0f);
		
		normal.z = sqrt(1.0f - dot(normal.xy, normal.xy));
		shade = saturate(dot(In.SunLightDir.xyz, normal.xyz));
		
		if (shade > 0 && use_parallax_shading)
		{
			height = (tex2D(NormalTextureSampler, In.Tex0).r);
			heightmap =  1.0f-height;		
			float startoffset = fHeightMapScale*In.SunLightDir.xy/max(0.2,In.SunLightDir.z);
			int numsteps=20;
			float step = heightmap*float(1.0/numsteps);
			float stepheight = 1.0f+.5*step;
			[unroll(numsteps)] while ( stepheight > height)
			{
				height = tex2D(NormalTextureSampler, In.Tex0+startoffset*(heightmap-step)).r;
				stepheight -= step;
			}
			if (saturate(stepheight)<height)
				Output.RGBColor.rgb *=0.5*(height-heightmap)+0.5;
		}
	
	}
	else
	{
	normal.xy = (2.0f * tex2D(NormalTextureSampler, In.Tex0).ag - 1.0f);
	
	normal.z = sqrt(1.0f - dot(normal.xy, normal.xy));
	shade = saturate(dot(In.SunLightDir.xyz, normal.xyz));
	}
	
	float4 total_light = vAmbientColor;//In.LightAmbient;
	if (PcfMode != PCF_NONE)
	{
		float sun_amount = 0.03f + GetSunAmount(PcfMode, In.ShadowTexCoord, In.ShadowTexelPos);
		total_light += shade * sun_amount * vSunColor;
	}
	else
	{
		total_light += shade  * vSunColor;
	}
	total_light += saturate(dot(In.SkyLightDir.xyz, normal.xyz)) * vSkyLightColor;
	
	#ifndef USE_LIGHTING_PASS
		total_light += saturate(dot(In.PointLightDir.xyz, normal.xyz)) * vPointLightColor;
	#endif

	Output.RGBColor.rgb *= total_light.rgb;
	Output.RGBColor.a = 1.0f;
	Output.RGBColor *= vMaterialColor;
	
	float4 tex_col = tex2D(MeshTextureSampler, In.Tex0);
	INPUT_TEX_GAMMA(tex_col.rgb);

	Output.RGBColor *= tex_col;
	Output.RGBColor *= In.VertexColor;

	OUTPUT_GAMMA(Output.RGBColor.rgb);

	return Output;
}

All this subscript thing is called swizzling, and is just mnemonics. HLSL allows you to refer to the individual components of float4-based variables as either

.x .y .z .w

or

.r .g .b .a

. Normally you use the first ones for positions and the second one for colors, but it doesn't really matter.

What this error is telling you is that the fourth component of that

float

variable, that you are isolating/referencing doesn't exist. Most probably, the world normal is stored as a

float3

instead of a

float4

, so you can only access its

.xyz

/

.rgb

components.

The good thing is that is pretty easy to correct. And I suppose that the original

struct

was altered to contain an additional modulator, because normals are three-dimensional, I don't know why a

float4

would be needed by default, unless you want to carry additional information from the vertex shader to the pixel shader. Normals are interpolated automatically by the GPU.

 

[La Grandmaster]

Thanks Swyter, decided to go a slightly different route and implement xenoargh's altered parallax version (http://forums.taleworlds.com/index.php/topic,208694.msg5240180.html#msg5240180), as I believe that does not require the green normal map textures to be altered, while Barabas' does? (not sure if this is correct at all).
Had to alter Xenoargh's version a bit but I think I've got it working, definitely looks better imo

Also I have been trying for the past hour but I cannot remove the random green hue effect you get on the terrain, Mark7 managed to remove it earlier in this thread, and I have been messing around with the vertex shader - vs_main_bump - for about an hour now but have not been able to remove the random green effect.

E.g what I want to do
Before

Spoiler

After

Spoiler

 

[Swyter]

Thanks Swyter, decided to go a slightly different route and implement xenoargh's altered parallax version (http://forums.taleworlds.com/index.php/topic,208694.msg5240180.html#msg5240180), as I believe that does not require the green normal map textures to be altered, while Barabas' does? (not sure if this is correct at all).
Had to alter Xenoargh's version a bit but I think I've got it working, definitely looks better imo

Also I have been trying for the past hour but I cannot remove the random green hue effect you get on the terrain, Mark7 managed to remove it earlier in this thread, and I have been messing around with the vertex shader - vs_main_bump - for about an hour now but have not been able to remove the random green effect.

E.g what I want to do
Before

Spoiler

After

Spoiler

Use it sparingly, as it is pretty taxing and people don't play M&B because of its pretty graphics. The green hue effect is provided by the engine when passing the terrain vertex colors to the shader. Just modulate the vertex coloring reducing its strength, so it isn't as noticeable with a brownier palette.

The easiest way is just to to do something like

vertex_colors_whatever *= 0.6f

in the vertex shader before they are outputted to the pixel shader.

But do it just for the terrain shaders, not everything that makes use of

vs_main

. Make a copy, something named like

vs_main_terrain

, replace in the techniques when needed and you are good to go.

 

[La Grandmaster]

@Swyter, Yeah I've created a terrain specific vertex shader already just to mess about with, thanks will give that ago 

 

[Jarvisimo]

Hello everyone,

I've got the shader source code set up and got it reloading in game.

I've done a lot of reading but some things about shaders still confuse me.

1. Are there any really simple or good tutorials out  there which explain the HLSL syntax like I'm five?
2. Is there a HLSL plugin for Notepad++ and if not, what do you guys use for code editing?
Found one!
3. This is a really long shot, but would it be in any way possible for someone to write some comments alongside a native M&B shader explaining exactly what each operation does (again, like I'm 5). I feel like this would really help me get started.

Thank you

EDIT: Also what is a 'standart' shader and why does it seem as though "Standard" spelt wrong so consistently?

 

[Swyter]

Hello everyone,

I've got the shader source code set up and got it reloading in game.

I've done a lot of reading but some things about shaders still confuse me.

1. Are there any really simple or good tutorials out  there which explain the HLSL syntax like I'm five?
2. Is there a HLSL plugin for Notepad++ and if not, what do you guys use for code editing?
Found one!
3. This is a really long shot, but would it be in any way possible for someone to write some comments alongside a native M&B shader explaining exactly what each operation does (again, like I'm 5). I feel like this would really help me get started.

Thank you

EDIT: Also what is a 'standart' shader and why does it seem as though "Standard" spelt wrong so consistently?

1.

http://http.developer.nvidia.com/CgTutorial/cg_tutorial_chapter01.html

And before you start lashing me because that's not what you want, read this: "The current version of DirectX at the time of this writing is DirectX 9, which includes HLSL, Microsoft's implementation of the same language syntax and constructs found in Cg."

2. I just use Notepad++ and set the syntax highlighting to C/C++. It works fine.

3.

http://http.developer.nvidia.com/CgTutorial/cg_tutorial_chapter02.html

4. They are conveniently Turkish. I don't know if its related with mtarini's standart shader, but it bothers me too.

 

[Jarvisimo]

That's fantastic, I'll take a look. Cheers Swyter.

I am trying to go very simple for now and get a desaturation shader working. But I'm clueless.

Never mind, I'm sure I'll get there eventually. please help

 

[La Grandmaster]

That's fantastic, I'll take a look. Cheers Swyter.

I am trying to go very simple for now and get a desaturation shader working. But I'm clueless.

Never mind, I'm sure I'll get there eventually. please help

This started off as a small message but hell its practically a mini tutorial now.
Note that the wording/phrasing in a lot of the following is most likely not technically correct as I am entirely self taught but this should help you out a bit.

HLSL & Desaturation ​

1) Desaturation is actually fairly easy once you know what you are doing, but getting your head round it is complicated, desaturation should obviously be done in the pixel shader not the vertex shader - so find the pixel shader you would like to desaturate.


Where to apply the code
2) Now we must decide what aspect of the shader do we want to desaturate and how we want to desaturate it. Do we want to desaturate the diffuse texture before normal map/specular effects are applied? Or do we want to simply desaturate the entire thing at the very end? ect.  In M&B there will be very little difference between the two as M&B does not feature colored specular maps, however if we just desaturate the diffuse we may still be left with vertex colouring - so you see we must decide what we want to desaturate and how we want to do it.

3) For the sake of simplicity we will just desaturate the entire thing (almost like a post effect after everything else has been done). Since it is after everything else has been done we go to end, you will see that every shader ends in

return Output;

this means that everything which is "saved" under the Output "prefix" will be given to the engine for rendering (this is not technically at all correct but it works and makes sense). 

So work back from here, we obviously want to apply the effect before it is given to the engine for rendering so we must add the code before return Output;   . So we go up, it is most likely that the next line up is OUTPUT_GAMMA(Output.RGBColor.rgb);  (for the sake of this tutorial we will ignore this and go up another line). The next line will most likely involved things being applied to the "Output" prefix. It may involve a fresnel effect being applied, or it may just be a texture being applied. Since we want to desaturate everything we want to add our code after this line - we now know where to add the code.


Applying the code
4) To make the actual desaturation effect obvious to you and clear we will take Output.RGBColor.rgb and copy it, we will then desaturate the copy, and then overwrite the original Output.RGBColor.rgb with our copy.

Ok so we first make a copy, to do this we create a new float3 (note that this is 3 because we want the 3 channels (r,g & b) of Output.RGBColor.rgb. If we also wanted the alpha we would use a float4. The following line shows the above

	float3 RGBColor_ALT = Output.RGBColor.rgb;

- I have set RGBColor_ALT to equal Output.RGBColor.rgb;

We now have our copy. And now we want to desaturate it - but how to do this. This is actually a lot more complicated than you would first think. The float we created above - RGBColor_ALT -  contains three rgb "channels", we could simply reduce the value of each of these, and this would surely reduce the colour - if we decrease the amount of red in a picture and the amount of blue and the amount of green we are surely reducing the colour. This will not work however as we are not reducing the colour we are in fact reducing the amount of light. If you open up a colour wheel tool online and reduce all three of these values you will notice that by simply reducing the three rgb channels we do not desaturate, instead we make the colour darker (reduce the amount of light)  (do it on this tool - http://www.colorspire.com/rgb-color-wheel/).  If we set RGBColor_ALT to 0, we would have a black colour as there would be no light (the 0) interacting with the model. If you look at the above tool, we do not want to reduce the colour on the y axis, we want to reduce it on the x axis - we want it to maintain its position on the y axis but move it to the very left on the x axis - sounds complicated but try this out in the above tool.

So how do we do this, we do this by taking into account how the human eye perceives light. We do this using the following,

float greyscale = dot(RGBColor_ALT.rgb, float3(0.30, 0.59, 0.11));   

- The float3(0.30, 0.59, 0.11) part refers to how we perceive the three channels of light, you will notice that it adds up to 1.
- The RGBColor_ALT.rgb refers to the copy we created earlier.

The line of code basically takes the dot product of 1) the three channels saved in RGBColor_ALT.rgb and 2) how we perceive each of these three channels. It then saves it the float greyscale. In all honesty you do not necessarily have to understand the above line of code, as long as you just understand roughly what it is doing - it is simply reducing the colour intensity of the 3 channels of light saved in RGBColor_ALT.rgb, and then saving it to float greyscale.

Final Code

        //rest of your code above here
        float3 RGBColor_ALT = Output.RGBColor.rgb; 

	float greyscale = dot(RGBColor_ALT.rgb, float3(0.30, 0.59, 0.11));    
	Output.RGBColor.rgb = greyscale;
	
	Output.RGBColor.rgb = OUTPUT_GAMMA(Output.RGBColor.rgb);


	return Output;
}

The above code should work

However you will notice that the  float3 RGBColor_ALT part is actually pretty useless ( I just added it to make things more clear). We could actually just take Output.RGBColor.rgb and directly perform the greyscale calculation on that (almost like cutting out the middle man - the middle man being RGBColor_ALT).

        //rest of your code above here

	float greyscale = dot(Output.RGBColor.rgb, float3(0.30, 0.59, 0.11));    
	Output.RGBColor.rgb = greyscale;
	
	Output.RGBColor.rgb = OUTPUT_GAMMA(Output.RGBColor.rgb);


	return Output;
}

EDIT: Note that the above completely desaturates everything, you can merge the desaturated version ("saved" in the float greyscale) with the non desaturated original ("saved" in Output.RGBColor) using the lerp command. This will give you a slight or strong desaturation effect depending on how you use it.  e.g

Output.RGBColor.rgb = (lerp(greyscale, Output.RGBColor.rgb, 0.5f)); 

Note this line replaces the line which says Output.RGBColor.rgb = greyscale; so would go directly before the output gama line.

the last value states how desaturated you want the final texture to be, the lower then the more desaturated it will be. 0.0f means  Output.RGBColor.rgb will be set 100% to greyscale, while a value of 0.5f means Output.RGBColor.rgb will be set to 50% greyscale and 50% the original coloured version.

Hope this helps

 

[Jarvisimo]

Marry me. YOU ARE A GOD.
I'll get straight to it tomorrow afternoon

 

[Jarvisimo]

Okay, I read through it more thoroughly and whilst I cannot thank you enough for how helpful that was, there are a few terms I am iffy on.

1. Why are variables called "floats"? What is floating? I understand about the different data types depending on the channels though.

2. What do you mean by the 'dot product'? Is the dot product the final colour of a pixel?

3. How would one apply a shader only in a specific scene in game? And would this greyscale affect UI etc?

Thank you so much for your time.

P.S Swy, this nvidia documentation certainly isn't light bedtime reading. Looks like I'll have to have another crack at understanding more tomorrow.

 

[La Grandmaster]

Okay, I read through it more thoroughly and whilst I cannot thank you enough for how helpful that was, there are a few terms I am iffy on.

1. Why are variables called "floats"? What is floating? I understand about the different data types depending on the channels though.

Stands for floating point, they are not variables in the same sense as those in the module system. Also they do not have to be floats, they can be integers, or halfs also (amongst other things). It depends what you are storing, integers have no decimal points (or rather decimal points all 0), halfs and floats do. Halfs are less precise than floats as they use less storage bits). Halfs and Floats can be 1 dimensional ("float" e.g a decimal number), 2d ("float2" e.g uv coordinates), 3d ("float3", e.g positional data x,y,z or r,g,b), 4d (float4 e.g x,y,z,w  or r,g,b,a (a being the alpha value)), they can also be matrices (imagine like a 4 by 4 table).

2. What do you mean by the 'dot product'? Is the dot product the final colour of a pixel?

You dont really have to understand it in this example. Its ussually used to work out a value which is roughly along the lines of how perpendicular 2 vectors are or the cosine of the angle between them - atleast this is my understanding. Obviously in our example we dont really have positional vectors though (x,y,z), we have color (r,g,b) which makes things harder to explain. Again I have never had any formal training beyond higher maths and GCSE computing so the actual technicalities and proper wordings/definitions are over my head. I just know what I want and roughly how to do it

If you are really interested the second post here explains it properly (http://stackoverflow.com/questions/6240804/help-understanding-the-use-of-dot-product-in-this-example)

3. How would one apply a shader only in a specific scene in game? And would this greyscale affect UI etc?

You are misunderstanding how shaders work, there are many shaders active in each scene (and on the world map). Applying greyscale to say the bump shader would affect all models using this shader, but would not effect models not using this shader. The UI uses very simple shaders of their own. If you want to apply greyscale to everything onscreen  you wuold have to do this post processing/ post_fx, which is quite different.

You can turn certain aspects of shaders on and off via the module_system though (how I have created seasons). This is how you would have greyscale active in certain scenes and not others

 

[Jarvisimo]

Simply brilliant, thanks a lot. I'll have a go in a few hours and get back to you

 

[Jarvisimo]

It worked!

Okay, I have fiddled around making that all work nicely at the desaturation I want.

Next step: Flag shader and getting that weird blue surround thing that Docm30 made.

So flag shader first. I'll try and have a go before falling at your feet begging for salvation, Grandmaster.

EDIT: More questions:

1. Why do some shaders end in return Out; whereas others end in return Output;
?

2. I keep running into the instruction limit when I try to add the desaturation. :/ Has this been fixed or will we always be limited to 64?

 

[La Grandmaster]

Out. is just a prefix, anything saved under this prefix will be "released/passed out" from the shader. It can be anything, doesnt have to be Out. or Output.  - I use Result. in many of my shaders. Although looking at the code, Out. tends to be used for vertex shaders and Output. for pixel shaders. I would strongly suggest you keep to this however as you cannot just simply change Out. to Return. and expect it to work


As for the instruction limit, the first few messages in this thread deal with this. You must find the techniques which use the pixel shader you have edited (just do a search). You must then change any techniques which use PS_2_0 to PS_2_X

but once I got around the 2.0 limit, it got a lot easier, lemme tell you- I spent a couple of hours just trying to free up a couple of instructions this morning 

As for the how... go look at the lines that say, "compile PS_2_0".  You just change them to PS_2_X, that's it. 

Same with the PS_2_a; it's pointless to specify that when you can just go straight to _b, and again, there practically aren't any cards with those specs that can even play the game with DX9 anyhow.

It also makes the FXOs compile faster 

Be careful though every line of code/calculation you perform decreases performance. Although I have some rediculously expensive shaders in my mod and dont have any massive performance impacts.