Flag Meanings (in materials, shaders, textures, meshes...)

[cmpxchg8b]

For all meshes? Even those which won't need them?

Yes, but it doesn't really matter. There is no difference in memory usage (all meshes use the same memory layout) and the difference in performance is probably insignificant and affects only the loading screen.

Ok for terrains, but what does identify a head mesh? How are they special?
BTW head meshes have pre-computed tangent dirs on them.

They are special because they are deformed and mirrored by the engine. Both normals and tangents are computed on the fly, I don't think the pre-computed ones do anything at all.

 

[mtarini]

Thanks for the info as usual!

Yes, but it doesn't really matter. There is no difference in memory usage (all meshes use the same memory layout) and the difference in performance is probably insignificant and affects only the loading screen.

I see.
But the overall picture comes out a little stange, right?
It seems that,
(1) TW decided to introduce optional pre-computed tangent dirs.
    Most meshes have that option on (it's mesh flag 0x10000).
    That costs as much as 13 extra bytes per vertex of disk space, i.e. some 28% of total on average.
    (plus it costed the hassle of changing the file format for them and making them optional)
    The only gain: avoid the time expenditure of recomputing tangent dirs on mesh-load.
(2) Then, for all meshes which don't actually need tangent dirs,
    they needlessly spend time to recompute tangent dirs on mesh-load

Put like this, it looks like one of these contradiction memes.
Maybe we are getting something wrong.

Both normals and tangents are computed on the fly, I don't think the pre-computed ones do anything at all.

mmm... I see, however, side note:
they might as well have used an interpolation of the normals and tangents defined in each frame, i.e. facemorph, just as they do with the xyz position.

That would look smarter because they would let artists determine which normals they want exactly (which is the same reason why normals are stored, not computed, in any other model).




Meanwhile, in Soviet Russia, there is another strange point in need of light shedding...

Material flag 0x10000 is currently labelled "Render 1st" (again, I've no idea where I got that from, originally).
Apparently, it means that the object is to be rendered before any other, except, I expect, similarly flagged ones.

I wonder if that is so, because it seems redundant, as it look just another "render order" value. They could have used minimal render order for that.

Unless that flags actually means "render even before some hardwired operation", like dunno... some pass or the initial clear-screen (ok, that cannot be it) or something.

So, any info about that which would be worth to be reported to OpenBRF users?

 

[cmpxchg8b]

No idea about the tangents, but given how many things are done by this engine I'm not sure there has to be a logical explanation.

mmm... I see, however, side note:
they might as well have used and interpolation of the normals and tangents defined in each frame, i.e. facemorph, just as they do with the xyz position.

I guess. Just normals, though, frames don't have tangent info.

Meanwhile, in Soviet Russia, there is another strange point in need of light shedding...

Material flag 0x10000 is currently labelled "Render 1st" (again, I've no idea where I got that from, originally).
Apparently, it means that the object is to be rendered before any other, except, I expect, similarly flagged ones.

I wonder if that is so, because it seems redundant, as it look just another "render order" value. They could have used minimal render order for that.

Unless that flags actually means "render even before some hardwired operation", like dunno... some pass or the initial clear-screen (ok, that cannot be it) or something.

So, any info about that which would be worth to be reported to OpenBRF users?

Nothing special about it, it just gets rendered before any other mesh (even the ones with the lowest render order). The fact that a mesh with the render 1st flag in the material doesn't get rendered in the depth pass makes me think it was made specifically for skyboxes.

 

[mtarini]

Small off-topic note:

frames don't have tangent info.

Actually, they do don't, at least judging by the mesh format I reverse engeneered (for the first versions of OpenBRF).
(And it makes sense that they do, they are aren't much needed)


MORE off-topic digression:

That data is stored in the usual "TW" way, the same used for normals and positions, which is a bit of a mess.

In details:

• for a mesh without vertex animation: positions and normals and tangents (if present) are stored as you would think,
  i.e., respectively: per "[pos]" (as the rigging), per "[vertex]" (together with UV-mapping, the color and tangents).
• if the mesh has frames (aka vertex animation): as above, plus each frame redefines all positions, and all normals, and all tangents for that frame.
  This means that the positions+normals+tangents originally defined for the static mesh are just a waste of space in this case.

edit: corrected, see posts below

It is like a good frame is wasted.
Long ago I decided that OpenBRF will show that "wasted" frame as the frame 0 of the vertex animation, because, well, the data is there after all and I didn't want to hide it.
But it is clear that the game just ignores that frame (in all cases when a vertex animation is used, which are quite a few, and include most armours).

 

[Swyter]

It is like a good frame is wasted.
Long ago I decided that OpenBRF will show that "wasted" frame as the frame 0 of the vertex animation, because, well, the data is there after all and I didn't want to hide it.
But it is clear that the game just ignores that frame (in all cases when a vertex animation is used, which are quite a few, and include most armours).

I've always wondered about this, thank you for clearing it up.

I believe that many years sharing the same codebase has messed up their idea of a coherent logic and they just hack stuff atop their previous games.

It's not a bug, the rgl engine is full of features. Thank them for those loopholes, quirks and undefined behaviors which allow us to do crazy stuff, like putting new HLSL techniques in front of the expected ones so they are loaded in 1.011.

Just discovered thanks to some people sharing their games on Youtube that on some gfx cards they show the rainbow instead.
Guess who has to make shaders optional until Warband. Stuff like this is what makes this hobby fun.

 

[cmpxchg8b]

I'm 100% certain that the engine stores just one set of tangents (the initial ones)... even the format doesn't support reading them per-frame (check BrfFrame::Load).



mtarini: you mean, in my own code? let me check... (can it be that I remember wrong?)

 

[mtarini]

You are 100% certain, and 100% right.
I remembered wrong!
(amended my last post above to avoid confusion)

As you say, the tangent dir aren't duplicated per frame...

(and TW is right on this choice: even tough, strictly speaking, each frame would need its own set, the original set is a good enough approximation, as long as the shapes don't vary too much between frames)

(It would not be a good approximation in presence of drastic differences between frames... like gloves or hands, where the frames are 1:[open-hand], 2:[fist/grip], and which, surely not by chance, aren't bumb-mapped at all).

(and, to continue on digressing farther and farther....
isn't it strange how the game seems only to ever use the closed hand frame? Open hand seems to be just a vestigial residue of some past choice...
if your script enhancer was still being mantained, this would be a good chance for a new cool operation, on the lines of "agent_set_hand_pose( agent_no, [closed|open] , [left|right|both] )" )

 

[mtarini]

(apologies for double posting)

I believe that many years sharing the same codebase has messed up their idea of a coherent logic and they just hack stuff atop their previous games.

It's not a bug, the rgl engine is full of features. Thank them for those loopholes, quirks and undefined behaviors which allow us to do crazy stuff, like putting new HLSL techniques in front of the expected ones so they are loaded in 1.011.

Just discovered thanks to some people sharing their games on Youtube that on some gfx cards they show the rainbow instead.
Guess who has to make shaders optional until Warband. Stuff like this is what makes this hobby fun.

You are so right.
Yet I would prefer not to have to pass thorough loopholes and quirks. I like it more when Armagan opens the main road to do things.
That said, I've my own share of quirk exploitation...
I think my most satisfying quirk-ride has been the one I did to "orc-ify" ("orc-ize"?) every animation in TLD,
in spite of the severe limitations on race-wise customization of the animation sets.
But I could mention many more

Link to the rainbow thingy? I couldn't find it on youtube.

 

[mtarini]

(and now it is a triple post  )


Forgive me [cmpxchg8b], I need to ask you for yet another piece of info, and yet another of your glances inside the game behavior...

Thanks for the patience!

[Material transparency function bits:]
...
7 (auto-select): if material color is transparent it will use mode 1, else 0

What do you mean by "material color"?
Maybe content of the diffuse texture (whether or not is has at least one texels with alpha<1)?
Per-vertex color, maybe? (Whether or not there is one with alpha<1)
Both options seem a bit unlikely...

 

[cmpxchg8b]

It's a per-mesh color that affects all vertices (vMaterialColor in mb.fx). I don't think it can be set it via .brf files.

 

[mtarini]

Oh, I see...

you are right in that there is no trace of it in brf files (mesh or materials), or in the python files...
so it seems not be controllable by modders.

Well, it must be some control that TW reserved to itself to be able to impose a given color achieve this or that effect.

Oh wait, mb.fx reveals that it is used in at least two occasions:
- to set the hair color (by multiplying the "blonde" color),
- to set the aged version texture overlay (which is not transparecy at all, naturally: it is just interpolating between two textures).

Oh, it would seem, also to set general color of light (reddish at sunset), however much "material color" is a misnomer in this case.

Plus, my guess: also used to highlight selected objects in scene edit mode, and hacks like that.

Cannot be used for red blood splatters, though, as these are evidently set per vertex (not per mesh).



Oh well, for what OpenBRF preview is concerned, value 7 will mean "opaque" (no alpha blending).

So thanks, solved, not I can release a new version with all that info in it.

 

[Somebody]

Noticed that there were still a bunch of Unknown texture flags - does the "4" flag common on a number of normal map/specular do anything important?

 

[Swyter]

And the vast majority of color operator slots from the shaders are undocumented. What we usually do is a bit of guesswork copy-pasting from configurations which are related with what we're trying to achieve.

Other than that. I think that the BRF format is pretty much done. At least for now.


Edit: I'm a jackass.

Not 100% sure, but almost.
By the way, if you were wondering what the colorop/alphaop stuff is (the lowest byte):
http://msdn.microsoft.com/en-us/library/ms886501.aspx

Still. I don't have a clue about that 100 in the flag section:

And a -1 in the map input. This one is from the second sampler on the map_ocean_shader_ffp shader:

 

[cmpxchg8b]

And the vast majority of color operator slots from the shaders are undocumented. What we usually do is a bit of guesswork copy-pasting from configurations which are related with what we're trying to achieve.

To be precise, it's not just that value I mentioned, but instead:

SetTextureStageState(stage, D3DTSS_COLOROP, opValue & 0x1F);
SetTextureStageState(stage, D3DTSS_COLORARG1, (opValue >> 5) & 0x7);
SetTextureStageState(stage, D3DTSS_COLORARG2, (opValue >> 8) & 0x7);

I guess it would be better if OpenBrf allowed you to set the three separately (combo box for op selection, combo box + flags window for arg selection).

Still. I don't have a clue about that 100 in the flag section:

The lowest byte of flags specifies the texture coordinate set index. I believe all shaders have this set to 0 (i.e. texture coords are the ones specified by vertex data), but for sake of completeness the values are:

0x00 -> SetTextureStageState(stage, D3DTSS_TEXCOORDINDEX, D3DTSS_TCI_PASSTHRU)
0x10 -> SetTextureStageState(stage, D3DTSS_TEXCOORDINDEX, D3DTSS_TCI_SPHEREMAP)
0x20 -> SetTextureStageState(stage, D3DTSS_TEXCOORDINDEX, D3DTSS_TCI_CAMERASPACENORMAL)
0x30 -> SetTextureStageState(stage, D3DTSS_TEXCOORDINDEX, D3DTSS_TCI_CAMERASPACEREFLECTIONVECTOR)
0x40 -> SetTextureStageState(stage, D3DTSS_TEXCOORDINDEX, D3DTSS_TCI_CAMERASPACEPOSITION)

The rest is "regular" flags:
0x100 -> uploads the texture offset transform to the fixed function pipeline (generated by the engine for stuff like water, in HLSL you would access something equivalent via texture_offset)

And a -1 in the map input. This one is from the second sampler on the map_ocean_shader_ffp shader:

-1 means simply that the stage receives no texture input

 

[mtarini]

'To make that "100" a bit less mysterious: it is in base 16 (0x100) and therefore is just 1<<16, or the 17th bit set.
The misunderstanding is my fault... ColorOp and AlphaOp are also bitmasks by the look of them (edit: as the post above details),
but I make OpenBRf write them in base 10.

As for the colorOp: they still are mysterious, because that table only refers the the last 8 bits of it.
For example that 1412 in your screenshot means code 24 (DOTPRODUCT3) and bit 0x400  (i.e. 1<<10, i.e. 1024).
And the meaning of last bit is mysterious. (edit: clarified by the post above)

My understanding is that this entire part can be safely disregarded beause it is only used when shaders aren't, which means basically never (and increasingly more so)


Longer explanation:

Before you had shaders, programming graphic effects was a little like playing with a closed box full of buttons and dials.
These controls let you customize what would happen when you sent in the triangles, and that's all the degrees of freedom you had.
It's the so called FFP (Fixed Functionality Pipeline): the hardwired flow the graphic card did.

For example, these "buttons and dials" would let you set whether the 2nd access to the texture was to be added OR multiplied OR .... , in a limited list of choices (that part is the table linked by xmpxchg8b). Other "buttons" would let you choose for example if specular components was to be computed, and countless other things.

In a modern approach, with shaders, many of these "switches" are gone: instead, you give a program (actually, a small set of programs, e.g. the vertex shader and the point shader) which describes exactly what happens, how, which order. It is done so that programmers are incomparably more in full control and can unleash they perverted fantasies when it comes to graphical effects.

A part of the "button and switches" remains: they control the stages which are left out of the programmable stages of the pipeline (for techincal reason, mostly to do with efficiency). For example, all the parts which are the concern of the Rasterizer (e.g. if back-facing faces should be culled or not), or of the Output Combiner (e.g. all which concerns the depth-test or the alpha blending... but not the alpha kill), or of the Texture Fetch (e.g. the clamping or repeating of texture coords,  usage or not of mipmapping, ...).


Why does any of that concerns modders?

A certain part of the settings in materials/textures/shaders and in flags instructs the game how to set up the FFP.
When shaders are used (i.e. basically always) , these bits have no effect.

Another part, by contrast, instructs how to set the switches which still have effect even when you use shaders.

I suspect the entire Texture Accesses settings of shaders belong to the 1st category.

 

[cmpxchg8b]

I ran a poll in the cRPG forums some time ago, and it showed that quite a lot of people are still running the game in DirectX 7 mode (which is actually mislabeled DirectX 9 FFP).
Perhaps those fields are not that useless yet...

 

[mtarini]

Interesting... link?

This is a matter of personal taste, but, as a modder,
I still would not be ready to pay too much a price for back compatibility with No-Shaders mode.
(see considerations below the line)

Our mod TLD wasn't: without my shaders, many things would come out seriously broken, and there was nothing we could have done short of dropping significant features for everybody. I can't recall this being, by itself, the cause of any trouble.
(I can remember an endless flow of troubles stemming out form the fact that in old M&B 1.011 customizing shaders would make the MOD installation a lot more complex, requiring to mess with potentially OS-protected native files, and if that failed the game would crash on strart-up -- that was came to be known as the "windy_flora" trouble. But that's unrelated)

That said, it would sure be supercool if OpenBRF had a better support for all settings, including FFP-only ones!
So the info above are useful.

Actually, now I release that OpenBRF should clearly mark what only has any effect in FFP!
E.g. by a short parenthesized formula "(FFP only)" appearing in status bar tips.
I wish I could start marking stuff in that sense, starting with material flags.
That would make life simpler to these modders who decide to disregard or postpone any problem linked to FFP (no-shader-support) behavior.




About the poll... it is surprising to know, but it is also a matter of why people run the game in Direct X 7.

* If the answer is "because they have to (their card doesn't support latest DirectX)",
then I (as a Modder) would ask myself if my mod is intended for such low spec computers.
Maybe winning these computers is a lost battle anyway, or a battle that would require giving up too many features.

* If the answer is "because the game runs too slow using Dx 11 for them",
then I (as a Modder) would worry about why that is the case. Maybe my shaders are too complex or not optimized enough.

* If the answer is "because the users are setting their game wrong" or "for some reason they didn't install last DX"
then I (as a Modder) would just write, in red gigantic font in the TroubleShooting section of my MOD:
"Doesn't work for <describe what goes wrong with FFP>? Install latest directX and set the game like this and that!"

 

[cmpxchg8b]

Interesting... link?

http://forum.meleegaming.com/general-discussion/do-you-play-with-dx7/

This is a matter of personal taste, but, as a modder,
I still would not be ready to pay too much a price for back compatibility with No-Shaders mode.
(see considerations below the line)

Oh, I agree. In our cRPG beta client I dropped support for "DirectX 7" entirely.