Please note: this tutorial assumes you know what a shader is and how to write your own, either in HLSL or in ARB fragment program. If that's not the case, there are plenty of tutorials on the web. Much more so in GLSL than in HLSL, but the two language are very similar and what you learn in one can be applied to the other.
1. Background: in-game shaders, in Mount&Blade/Warband
(not to be confused with "preview shaders" in OpenBRF, the ones used to preview your meshes within openBRF. They have nothing to do with what the game uses, even if they are designed to produce similar results. Preview-shaders are written in GLSL, in-game shaders either in HLSL or in ARB-fragment-program language)
M&B/WB vanilla uses a set of shaders, but you mod can, with some effort, add new custom ones written by you to the list.
As you can see looking at vanilla files, BRF files have in them items of a category called "Shaders" (along with: Materials, Textures, Meshes, Skeletons, etc). So you might think that the shader itself is kept inside the brf file.
In reality the situation is like for Textures: a "Texture" item inside a BRF file is not the image itself: it is just a "link" pointing to a .dds file outside the BRF, plus a few additional metadata like flags etc. The actual data (in that case, texels) is kept in the .dds files.
Similarly, a "Shader" item (inside a BRF file), is merely a link to actual data (in this case, shader code) which is kept outside the .brf file, plus a few additional data like flags. In this case the actual code being pointed by the item resides in one (or two in WB) global files called "mb*.fxo" (for most of them: fewer of them are kept in separate ".pp" files).
(Note that for most other categories of BRF items that's not the case: e.g. a Mesh item contains the actual mesh 3D data, an Animation item embeds all bone-rotations and timings, a CollisionMesh item contains all the 3D data, and so on)
The file(s) "mb*.fxo", part of the M&B installation, is located in the in game root folder, or optionally the mod root folder (in pre-WB M&B, it has to be the game root folder). It stores a collection of pre-compiled "techniques". Here, a technique being basically the union of a vertex program and a point program (aka fragment program, aka fragment shader). The shader-item points to a technique by name matching: a shader-item, regardless of its name, points to the "technique" specified in the field "technique", which in OpenBRF you can read (and edit) inside the Data box (in the mid panel) when you select that Shader item.
The "fxo" file(s) is a binary file and as such it cannot be directly edited in any easy way. Talesworld obtained that file as a compilation of a hi-level HLSL source file, and you can do the same to obtain your modified file. So to add a new shader you need to get that source file, edit it, and recompile it.
2. How to: add a new shader to your Mod
New shaders can be written in either ones of two languages: [HLSL] or [ARB] (follow the links for more info on them).
- [HLSL] = the high level shading language of DirectX, with "techniques" in it
- [ARB] = the low level, shared assembly like languages of GPU cards, with commands looking like three letters uppercase like "SUM" or "MAD".
ARB are more widely supported by low-spec systems, so that might be a good option for "fall back" shaders. However, if you use ARB the game will allow you to customize only the fragment shader, not the vertex shader.
If you want to use HLSL (suggested)
Step 1: grab hold the source code of vanilla shaders. M&B uses will find theirs in the game root folder: it's the file "mb.fx". WB users will find theirs here: [link]
Note: TW didn't have to share or ship that file. The game could have been sold "closed box" and would work just as well in game, only you would not be able to modify it. TW would have had all advantages not sharing their source code, but they did for the advantage of modders, so prise to Armagan and team.
Step 2: edit source and add your new fragment shader, and/or your new vertex shader, and a technique putting them together (with each other or with vanilla shaders). That's the core part, where you actually write new effects. This is not the place to teach shader programming (beginners can find tutorials on the web). The only relevant thing here is how you choose to name the final technique. Let's say you name it "foo".
Step 3: compile the modified shader source. This produces the file mb.fxo for your mode.
To compile the shader you can use any tool, for example the free Fx Batch Compliler (warning: requires .net framework installed).
A good side of this way of doing stuff is that most errors you can make in Step 2 are detected here in Step 3 as compliation errors, before even running the game.
Step 4: Place the fxo file you just produced where the game expects it to be found. WarBand users: it's in the "data" folder of your module. Old Mount&Blade: in the Game root folder. In the latter case, you'll have to substitute the original .fxo with your new version (backup, naturally). That's bad, because it will affect the game itself and all modules, not only yours. This will complicate your life when it will be the time to distribute your module. See "Pitfalls" later to see possible solutions.
If you want to use ARB
Step 1: Write your fragment program in a .pp file, name that file it in any way. Say you name it "foo.pp". Place that file next to the game exe, in mod root folder (pre-WB M&B: game root folder). Go directly to step 5.
Then:
Step 5: Add a new "shader-item" in OpenBRF. Name that new shader-item anyway you like (not necessarily as the technique). Best way to add it is to replicate an existing one, better if one similar in functionalities to the new custom shader (e.g. one performing the same number of texure accesses), so there are less flags to specify. There are several flags to specify, like the ones relative to texture access(es), or whether or not your shaders feeds on "rigging data", or whether or not it needs "tangent direction" data be sent to it. This is not the place where to describe what any of that means. Again: in case of doubt, your best bet is to replicate an existing shader to start with. One other thing to specify is the "fallback" field, that is the name of the shader-item (note: not the technique) which the game should use instead of this shader when this one is not supported (e.g. because it needs DirectX 9, like specified in its flags, and the system uses DirectX 7)
Step 6: specify that your new shader item uses the technique you made: in the example above, "foo", or "foo.pp" in case you used ARB.
Step 7: make the appropriate material-item use the new shader-item. Sets the rest of the data of the material-item according to need (e.g. which textures are used, flags, whether or not blending is activated, etc -- note that blending takes places or not accroding to the material flags, the shader doesn't do that: the shader can only specify which value of alpha is output for a pixel, not what it is done with that alpha -- e.g. blend or kill). Finally make sure the appropriate meshes use that material.
Step 8: Test it! Load the game and see how it looks. Important trick: when you re-edit a shader, you don't have to reload the game see the effects. Just don't quit the game, alt-tab out, repeat steps 2-4 above, alt tab back in. If changes don't kick in, then switching to full screen and back (alt-enter) should do the trick.
Optional step: preview in OpenBRF: Now that you have the shader in game, you might want to make a GLSL version of it so that OpenBRF can use it to preview the meshes using the new shader. GLSL is similar enough to HLSL that doing so should be easy. If you do, add your new preview-shader in "customPreviewShaders.xml" in OpenBRF folder, manually with a text editor (follow the example, should be easy).
3. Pitfalls and solutions
Distributing a Mod using custom shaders (pre-WB M&B only)
A problem with M&B is that the files you produced with steps 1-4 (or 1, with ARB), which are now necessary to your mod (the game won't even load without them), are specific to your mod but resides outside module folder, in the game root folder. M&B only has this problem, WB can store it in module root folder. A way to solve it is to ship your module bundled with the IronLaucher by [Swyter], whose purpose is to automatically replace a few vanilla files before running a mod (and rolled them back after). Be warned that in some system IronLauncher must be run "as administartor" in order to work correctly.
Shader-loading game bug
(more info needed, please share if you have them). In my experience, in M&B there is a recurring bug where new custom shaders are not loaded at game start-up (not sure about WB). They will be loaded if you just switch to fullscreen or back anytime after executing the game. A way to make this bug go away is to enlist your shaders not just in any BRF files but in the "core" files, specifically in "core_shaders.brf", a file in commonRes folder which is loaded even if you don't specify it in module.ini. This will give you another instance of the problem above, because you make your Mod rely on files which are not in module folder. The solution is like above: IronLaucher.
1. Background: in-game shaders, in Mount&Blade/Warband
(not to be confused with "preview shaders" in OpenBRF, the ones used to preview your meshes within openBRF. They have nothing to do with what the game uses, even if they are designed to produce similar results. Preview-shaders are written in GLSL, in-game shaders either in HLSL or in ARB-fragment-program language)
M&B/WB vanilla uses a set of shaders, but you mod can, with some effort, add new custom ones written by you to the list.
As you can see looking at vanilla files, BRF files have in them items of a category called "Shaders" (along with: Materials, Textures, Meshes, Skeletons, etc). So you might think that the shader itself is kept inside the brf file.
In reality the situation is like for Textures: a "Texture" item inside a BRF file is not the image itself: it is just a "link" pointing to a .dds file outside the BRF, plus a few additional metadata like flags etc. The actual data (in that case, texels) is kept in the .dds files.
Similarly, a "Shader" item (inside a BRF file), is merely a link to actual data (in this case, shader code) which is kept outside the .brf file, plus a few additional data like flags. In this case the actual code being pointed by the item resides in one (or two in WB) global files called "mb*.fxo" (for most of them: fewer of them are kept in separate ".pp" files).
(Note that for most other categories of BRF items that's not the case: e.g. a Mesh item contains the actual mesh 3D data, an Animation item embeds all bone-rotations and timings, a CollisionMesh item contains all the 3D data, and so on)
The file(s) "mb*.fxo", part of the M&B installation, is located in the in game root folder, or optionally the mod root folder (in pre-WB M&B, it has to be the game root folder). It stores a collection of pre-compiled "techniques". Here, a technique being basically the union of a vertex program and a point program (aka fragment program, aka fragment shader). The shader-item points to a technique by name matching: a shader-item, regardless of its name, points to the "technique" specified in the field "technique", which in OpenBRF you can read (and edit) inside the Data box (in the mid panel) when you select that Shader item.
The "fxo" file(s) is a binary file and as such it cannot be directly edited in any easy way. Talesworld obtained that file as a compilation of a hi-level HLSL source file, and you can do the same to obtain your modified file. So to add a new shader you need to get that source file, edit it, and recompile it.
2. How to: add a new shader to your Mod
New shaders can be written in either ones of two languages: [HLSL] or [ARB] (follow the links for more info on them).
- [HLSL] = the high level shading language of DirectX, with "techniques" in it
- [ARB] = the low level, shared assembly like languages of GPU cards, with commands looking like three letters uppercase like "SUM" or "MAD".
ARB are more widely supported by low-spec systems, so that might be a good option for "fall back" shaders. However, if you use ARB the game will allow you to customize only the fragment shader, not the vertex shader.
If you want to use HLSL (suggested)
Step 1: grab hold the source code of vanilla shaders. M&B uses will find theirs in the game root folder: it's the file "mb.fx". WB users will find theirs here: [link]
Note: TW didn't have to share or ship that file. The game could have been sold "closed box" and would work just as well in game, only you would not be able to modify it. TW would have had all advantages not sharing their source code, but they did for the advantage of modders, so prise to Armagan and team.
Step 2: edit source and add your new fragment shader, and/or your new vertex shader, and a technique putting them together (with each other or with vanilla shaders). That's the core part, where you actually write new effects. This is not the place to teach shader programming (beginners can find tutorials on the web). The only relevant thing here is how you choose to name the final technique. Let's say you name it "foo".
Step 3: compile the modified shader source. This produces the file mb.fxo for your mode.
To compile the shader you can use any tool, for example the free Fx Batch Compliler (warning: requires .net framework installed).
A good side of this way of doing stuff is that most errors you can make in Step 2 are detected here in Step 3 as compliation errors, before even running the game.
Step 4: Place the fxo file you just produced where the game expects it to be found. WarBand users: it's in the "data" folder of your module. Old Mount&Blade: in the Game root folder. In the latter case, you'll have to substitute the original .fxo with your new version (backup, naturally). That's bad, because it will affect the game itself and all modules, not only yours. This will complicate your life when it will be the time to distribute your module. See "Pitfalls" later to see possible solutions.
If you want to use ARB
Step 1: Write your fragment program in a .pp file, name that file it in any way. Say you name it "foo.pp". Place that file next to the game exe, in mod root folder (pre-WB M&B: game root folder). Go directly to step 5.
Then:
Step 5: Add a new "shader-item" in OpenBRF. Name that new shader-item anyway you like (not necessarily as the technique). Best way to add it is to replicate an existing one, better if one similar in functionalities to the new custom shader (e.g. one performing the same number of texure accesses), so there are less flags to specify. There are several flags to specify, like the ones relative to texture access(es), or whether or not your shaders feeds on "rigging data", or whether or not it needs "tangent direction" data be sent to it. This is not the place where to describe what any of that means. Again: in case of doubt, your best bet is to replicate an existing shader to start with. One other thing to specify is the "fallback" field, that is the name of the shader-item (note: not the technique) which the game should use instead of this shader when this one is not supported (e.g. because it needs DirectX 9, like specified in its flags, and the system uses DirectX 7)
Step 6: specify that your new shader item uses the technique you made: in the example above, "foo", or "foo.pp" in case you used ARB.
Step 7: make the appropriate material-item use the new shader-item. Sets the rest of the data of the material-item according to need (e.g. which textures are used, flags, whether or not blending is activated, etc -- note that blending takes places or not accroding to the material flags, the shader doesn't do that: the shader can only specify which value of alpha is output for a pixel, not what it is done with that alpha -- e.g. blend or kill). Finally make sure the appropriate meshes use that material.
Step 8: Test it! Load the game and see how it looks. Important trick: when you re-edit a shader, you don't have to reload the game see the effects. Just don't quit the game, alt-tab out, repeat steps 2-4 above, alt tab back in. If changes don't kick in, then switching to full screen and back (alt-enter) should do the trick.
Optional step: preview in OpenBRF: Now that you have the shader in game, you might want to make a GLSL version of it so that OpenBRF can use it to preview the meshes using the new shader. GLSL is similar enough to HLSL that doing so should be easy. If you do, add your new preview-shader in "customPreviewShaders.xml" in OpenBRF folder, manually with a text editor (follow the example, should be easy).
3. Pitfalls and solutions
Distributing a Mod using custom shaders (pre-WB M&B only)
A problem with M&B is that the files you produced with steps 1-4 (or 1, with ARB), which are now necessary to your mod (the game won't even load without them), are specific to your mod but resides outside module folder, in the game root folder. M&B only has this problem, WB can store it in module root folder. A way to solve it is to ship your module bundled with the IronLaucher by [Swyter], whose purpose is to automatically replace a few vanilla files before running a mod (and rolled them back after). Be warned that in some system IronLauncher must be run "as administartor" in order to work correctly.
Shader-loading game bug
(more info needed, please share if you have them). In my experience, in M&B there is a recurring bug where new custom shaders are not loaded at game start-up (not sure about WB). They will be loaded if you just switch to fullscreen or back anytime after executing the game. A way to make this bug go away is to enlist your shaders not just in any BRF files but in the "core" files, specifically in "core_shaders.brf", a file in commonRes folder which is loaded even if you don't specify it in module.ini. This will give you another instance of the problem above, because you make your Mod rely on files which are not in module folder. The solution is like above: IronLaucher.
(Or so it seems. I used Xenoargh's custom shaders and they work while simply in the mod's folder)
