New Module System Compiler Development

[Lav]

Please proceed to new thread for downloads.​

So, a few days ago I suddenlyTM got interested in the Module System compiler. Again.

Hopefully, the results speak for themselves:



So far it's very similar to swysdk, except it's pure Python. So it's somewhat slower than swysdk binary, but roughly 5 times faster than Native compiler.

As can be seen on screenshot, it supports object-oriented style for variable and entity naming, while retaining full compatibility with Native-style reference and variable names.

It also supports dynamic expression calculation. So you can easily write (assign, reg0, mesh.banner_a1 + 2) in your code and leave it for compiler to figure out what the resulting value will be. It is planned (and 90% of the underlying framework is already done) to also add support for dynamic runtime expressions - i.e. generating code dynamically from mathematical expressions using Module System variables, registers and slots. Those who remember WORMS will know what I'm speaking about, those who don't can find it through the links in my sig.

Compiler tries as best as possible to generate the same output as the Native compiler - up to the last space. There are a few exceptions, however.

First, it does not include entity tags into entity references. As far as I can tell, they are a legacy feature from Mount&Blade and Warband ignores them entirely. I may be wrong here, but that's what testing is for.

Second, when there are records with a duplicate reference name, Native compiler will match the name with the first occurence, my code will match with the last.

Third, I have removed several limitations that were imposed by compiler but do not actually exist in the game itself. So now it's perfectly possible to have item weight as much as necessary with any arbitrary precision (Warband will only display weight up to 0.1 unit). It is also possible to have huge quantity values for ammo and food items.

As a side note, Native compiler has a minor packing bug, where values for swing damage will inevitably overflow into item HPs. It's not important, since there are no items in the game for which both swing damage and health would matter, but may cause problems if a careless modder makes an item entry with superfluous damage definitions.

Oh, and the compiler has no use for ID_* and process_ files. Everything needed to compile the module fits into just two scripts. Of course, module files need some preparation for this to be working. In theory, compiler should work with raw Native module system without any modifications to the files whatsoever, but that theory has not yet been confirmed by practise, so there. Still, it's something that can and most likely will be achieved.

Planning to release first alpha version for public scrutiny once I get rid of one minor mistake in conversation.txt generation and finalize the export of quick strings and dialog states.

 

[Ra'Jiska]

After reading what you said, it sounded awesome in my minds.
But, I didn't get what it would do better than the original compiler.

Good luck though !

 

[Lav]

After reading what you said, it sounded awesome in my minds.
But, I didn't get what it would do better than the original compiler.

Good luck though !

1. Faster.
2. Easier configurable/extendable. I'm making it with compile-time script generation in mind, to allow the run-time scripts access to various parameters and values normally unavailable through the Module System.
3. Compile-time expressions.
4. Run-time expressions (AKA dynamic code generation).
5. Less quotes to print when modding. If it's quoted - it must be a string.
6. Automatic referencing between different modules without importing tons of ID_* files, no forcing the modder to compile his mod twice in several cases, no compiler failing when modder added a new icon, and without compiling his mod first used it for a party.
7. Slightly extending the boundaries of what can be compiled.

Nota bene. First test run of a fully compiled module resulted in game crash. I suspect that those tags are needed after all, so will have to implement them. Oh bother...

Update. It's even worse than I thought - there are places where the game expects simple index, and there are places where it expects index masked by entity tag. So it seems I'll have to crawl all over the compilation sequences again, manually checking each entry to ensure that correct tagged/untagged version is substituted.

 

[cmpxchg8b]

I suspect that those tags are needed after all, so will have to implement them. Oh bother...

The only tags needed by the game are register, local variable, global variable, string, and quick string. The rest are meaningless and never even checked.

Nice work, by the way.

 

[Lav]

I suspect that those tags are needed after all, so will have to implement them. Oh bother...

The only tags needed by the game are register, local variable, global variable, string, and quick string. The rest are meaningless and never even checked.

Nice work, by the way.

But something does cause my compiled module to fail. And if it's not tags, then I can only conclude that Warband's module parser is that sensitive. Because apart from tags, the only difference I can find between the output of my compiler and Native's is 0.000000 instead of 0.0 in some files and a few variations of spacing (having two spaces between values instead of one, or no space at the end of line where Native does have one). Quick strings and global variables look absolutely identical. There are some minor variations in dialog_states, but not in the hard-coded section, so it should be fine, I think.

Oh well, looks like I'll have to check everything file-by-file, copying my version over Native to detect which one if causing the game to crash. So far I already know it's not actions.txt.

 

[Lav]

Oh damn. All those crashes were actually because I did forget to include one value in the export line to dialogs. On the other side, considering the amount of export strings, one could consider that I forgot to include only one value.

Anyway, it works now and compiles the code and Warband starts up correctly. In fact it should even be 100% compatible with Native savegames - all references and globals have retained their values.

Here's the source for those who are interested. There's still a lot of work to be done - exception handling still does not provide enough information for my liking, even if it's already often more precise when pointing out the error. And the code is a horrible mess (as befits it's alpha status).

Module sources included in the package have been adapted to work with my compiler. That means no ID files and no referencing game entities directly - it's either class.name or "class_name", but not class_name. It's also slightly slower than in earlier tests as it has to parse all those string references instead of letting Python module loader do the job.

 

[Lav]

I'm making a small pause in development of the compiler as to be frank I'm mentally exhausted a bit at the moment. So until I resume the work on this project, it's probably high time to collect community opinion and the ideas on the priorities and development direction. What follows below is the list of all ideas I've had so far, and at least some of these ideas are quite likely to make it into the final product. Community opinion can, however, change the order in which the features are implemented and might even add some totally new ones which I haven't considered. So please voice your opinion, because it actually matters. Well, maybe.

1. Copy&Compile. It seems to be perfectly possible to make this compiler fully compatible with vanilla module files, so that the only thing one needs to do is copy the compiler files into the module folder and run the compiler.

2. ID_* files. I have excluded all dependencies on ID_* files at this stage, however for backwards compatibility it might make sense to generate them anyway. Ties nicely with the previous item, and is the prerequisite for the next.

3. Savegame compatibility tracking and enforcing. Make the compiler use the ID_* files left from the previous compilation to check for savegame compatibility. So if some previously existing reference suddenly changes it's value - the compiler should be capable of warning the modder and possibly re-ordering the entries in the module file to enforce the savegame compatibility. This may in turn result in broken continuous entity ranges, but it's possible to write heuristics that will detect this happening. Overall, this is quite a complex task.

4. Dynamic code generation. Leave to the compiler the boring task of composing the operation sequences to calculate complex mathematical expressions. Allow the modder write something like (store_add, reg7, g.var1 + g.var2, g.var3 * (g.var4 + g.var5)) and the compiler will generate something like:

(store_sub, ":tmpvar1", "$var1", "$var2"),
(store_add, ":tmpvar2", "$var4", "$var5"),
(val_mul, ":tmpvar2", "$var3"),
(val_add, ":tmpvar1", ":tmpvar2"),
(assign, reg7, ":tmpvar1"),

5. Extra data access during runtime. There are many properties and values which might be of interest to the module code during runtime that is only available to the compiler and the engine. It is perfectly possible however (and has been done numerous times with the vanilla compiler) to collect that data during compilation and inject this information into compiled code (usually in "script_game_start"). It would make sense to do this systematically and for all information that might be of use to the modder. Any loss of performance is irrelevant here as any code in "script_game_start" is only executed once (and the time spend is negligible compared to the time spent to run the game at all).

6. Extra custom data per entity. Similar to above, but instead of saving actual parameters and values, allow the modder to generate any extra data for the game entities. This allows for the possibility to move a lot of data hardcoded in "script_game_start" and related scripts into the module system, fully transparently for the modder. It would be nice, for example, to store family relationships in the module_troops.py file, instead of generating them in the code, and will allow a lot more possibilities for mod team members without scripting knowledge.

7. Extra entities. Add generation of some files which are normally not generated by the compiler. The prime example is Data/item_modifiers.txt. It would make a lot of sense to make it generateable by the compiler from the contents of module_item_modifiers.py. There are a few other files that might be added as well.

 

[Somebody]

Don't forget the extra module_data stuff.

 

[Lav]

Don't forget the extra module_data stuff.

That's item 7 in my list, yeah.

Okay, no new ideas posted, so I'll be starting with what I already wrote, that should be good enough for a start.

 

[Vornne]

This is interesting to me mainly because I have done similar things with the module system build scripts for PW (keeping Python but removing ID_*.py files, drastically reducing build times, adding some functions for compile time calculations and generators), though in a more gradual and piece meal way, not really a clean architectural rewrite. When originally profiling the code (using "python -m cProfile script.py") I found that the main performance increase came from removing the lower() function calls when dealing with identifiers, basically replacing it with an initial check that failed the build process immediately if an invalid (not lowercase, numeric and underscore) identifier was found; I don't know if that is useful information for this project.

When I tried this link to have a curious look at how you have implemented it, a login page was presented: do I actually need to have a Google drive account to see it, or is there a mistake in the link?

Here's the source for those who are interested.

https://drive.google.com/file/d/0BwIpJy6hoV-KNTE4UlBzNXBEeDQ/edit?usp=sharing

 

[Lav]

Damn, I thought I made all my Warband files public on the Google Drive. Fixed that, sorry.

I haven't profiled my code as even with full compilation, more than half of the time is still spent on simply loading the module files. Though what you say about lower() being inefficient is interesting, I'll need to check it.

 

[Lav]

Okay, the generated code finally seems to be 100% correct and playable.

Performed the test for "out of the box" compatibility. Copied my files to a Native Module System without making any changes to it and compiled successfully. Item 1 in the list checked off.

Saved game in Native and reloaded it after using my compiler. Successful.

Tried to play multiplayer with code generated by my compiler. Successful.

Tested dynamic code generation. Also works. There are ways to optimize the generated code, and I still need to implement support for a few more operations, but the general framework works correctly. That's item 4 off the list (or at least off the high-priority tasks list).

For those who are interested: Current compiler version (0.2 alpha) on Google Drive.

UPDATE: I still don't recommend actually using it. I've started using the compiler for my actual projects and already fixed a number of critical bugs. After all, no testing will ever compare to real life operation.

All things considered though, the next version should finally be usable in real projects, though I would still advise to wait until I finish the compilation error handling arc. Where it works properly, it's already much more detailed than the Native compiler, but that's the problem - it doesn't always work as intended, and more often than not, critical error-related information gets lost.

 

[Lav]

Regarding extra module files. I see the following candidates:

module_item_modifiers.py - converted to <module_folder>/Data/item_modifiers.txt

Includes the list of all item modifiers. Allows the modder to change their ID, name, price and rarity values. Does not allow to add new ones. Gameplay effects are listed in the file as comments and not compiled (as they are hard-coded).

module_flora_kinds.py - converted to <module_folder>/Data/flora_kinds.txt
module_ground_specs.py - converted to <module_folder>/Data/ground_specs.txt
module_skyboxes.py - converted to <module_folder>/Data/skyboxes.txt

Not sure these three are necessary - for like 99.9% modders the files will just clutter the module folder. Perhaps better to just copy them as they are into a subfolder in the module source folder, so they are always accessible.

module_hints.py - converted to <module_folder>/languages/en/hints.csv

Also num_hints parameter in module.ini should be automatically modified.

module_ui_strings.py - converted to <module_folder>/languages/en/ui.csv

Not all entries in ui.csv file are supposed to be modified, but many can. Maybe only allow some strings to be modified while keeping the rest intact?

module_info.py - converted to module.ini

It would make sense to move some parameters from module.ini into module_info.py file so as to make them directly accessible for module editor.

And regarding the extra module data that's only accessible during compilation time even though it would make a lot of sense to make it available in run-time:

module_items

• Associated item (for next_item_as_melee flag)
• Item itp_* constants
• Item itc_* constants
• Item imodbit_* constants
• Item associated factions (if any)
• Item extra parameters: weight, abundance, head/body/legs armor, speed, projectile speed, swing/thrust damage and type, hit points, difficulty, accuracy etc

module_scene_props

• scene prop flags

module_scenes

• scene flags (are they of any use?)
• dimensions (these should be available during mission time anyway though)
• water level
• list of accessible scenes (probably not, as this feature is officially deprecated)
• list of chest troops associated with this scene

module_skills

• skill name (as there's no str_store_skill_name operation in Module System)
• skill flags (only sf_inactive and sf_effects_party seem to be of any real interest)
• skill max level

module_troops

• troop flags (tf_guarantee_*, tf_inactive)
• troop scene and entry point (if any are assigned)

There are plenty of issues though, as the objective is not just to store all that information in some predefined troop's slots, but also update it as necessary if any additional entries are added to module files in later versions of the mod. Also would it make sense to create some fictional operations to retrieve this information (like str_store_skill_name) or provide the modder with some predefined scripts, or just with a few kind words of guidance?

 

[Lav]

Okay, since I've started using the compiler in my own projects, development speed for compiler itself will slow down from now on.

As for features, amount of feedback has been minimal, so I'll be assigning priorities according to my own left foot's expert opinion.

Namely, next feature's going to be a plugin system, allowing the modder to create a single plugin file for an individual feature instead of making twenty modifications and additions scattered around the Module System. And external constants will still have to go into module_constants. Any internal constants, any new entity declarations and possible code injections will be contained in plugin file, which will then be referenced in module_info, loaded by compiler and parsed/injected as necessary.

 

[Lav]

Plugin functionality mostly done.

Supported:

1. Dynamic extension of all game entity lists (troops, items, dialogs etc).
2. Dynamic injection of entity entries, entry data and code.

Still planned:

3. Dynamic overriding of existing game entries, either partial or complete. So plugin author could, for example, dynamically add a trigger to a vanilla item without modifying module_items.py at all.
4. Inter-plugin dependencies support.

Version 0.3 alpha on Google Drive.

Download includes my Companions Overseer presentation converted to plugin format as an example. Also modified Native module_dialogs.py file to see how injection works, and modified Native module_info.py to see how plugins are loaded. Plugin compilation has been tested and confirmed to work correctly on "out-of-the-box" Native Module System.

Note that current state allows "copy&compile" approach with Native Module System, however since my compiler doesn't yet generate ID files on demand, subsequent compilations may prove faulty. This is the only reason why the compiler is still considered an alpha-release. For as long as the Module System does not have any dependencies on ID_* files, compiler is fully functional.

Next release is expected to be a beta.

 

[Arch3r]

You're doing awesome stuff, but to 99% of the modders these things probably sound like foreign language. Don't let a seemingly lack of interest stop you, though. Once the modders see examples of the benefits of this compiler, surely they'll get a lot more interested.

3. Dynamic overriding of existing game entries, either partial or complete. So plugin author could, for example, dynamically add a trigger to a vanilla item without modifying module_items.py at all.

This seems really interesting. I have several ideas in mind for which I could really use this, keep up the good work!

 

[Lav]

You're doing awesome stuff, but to 99% of the modders these things probably sound like foreign language. Don't let a seemingly lack of interest stop you, though. Once the modders see examples of the benefits of this compiler, surely they'll get a lot more interested.

Don't worry, I'm perfectly aware of how many qualified programmers are present here. And my motivation currently comes from the fact that I'm using this compiler for my live projects (which is actually why plugins appeared before ID-files generation - my projects have been completely converted to OOP-style identifiers, so I no longer need ID-files at all, and as such this task has lower priority for me).

3. Dynamic overriding of existing game entries, either partial or complete. So plugin author could, for example, dynamically add a trigger to a vanilla item without modifying module_items.py at all.

This seems really interesting. I have several ideas in mind for which I could really use this, keep up the good work!

And if you could provide use case examples, that would really help.

Currently, I'm considering something like:

override(itm.horse_meat, [None, None, None, BASE | itp_merchandise, None, None, None, None, imodbits_food])

Or, alternatively:

override(itm.horse_meat, None, None, BASE | itp_merchandise, None, None, None, None, imodbits_food)

Essentially, we tell the compiler to override item with identifier "horse_meat", and we override two parameters: add a merchandise flag (BASE is obviously whatever value was there before the override) and a set of food-related imodbits (assuming there are some). All other fields are untouched (None).

However this is clearly inadequate for more advanced tasks. For example, someone might need to make certain modifications to all one-handed weapons, including those declared in other plugins. And yeah, it's perfectly possible and I can roughly imagine how the compiler would perform such a task. But this would require a more advanced syntax, so plugin developer could provide a filter that entity list would be iterated with. Perhaps even a custom callback in the most extreme of cases.

 

[Arch3r]

3. Dynamic overriding of existing game entries, either partial or complete. So plugin author could, for example, dynamically add a trigger to a vanilla item without modifying module_items.py at all.

This seems really interesting. I have several ideas in mind for which I could really use this, keep up the good work!

And if you could provide use case examples, that would really help.

I was thinking along the lines of setting item names to match randomized faction names (overriding item names should be possibly right?) as well as having an item act differently (different trigger called, possibly changing models if that would be possible) depending on which team the player wielding it is on.

Especially overriding the model could be really useful, I can see people using it for damaging equipment.

 

[Lav]

I was thinking along the lines of setting item names to match randomized faction names (overriding item names should be possibly right?)...

Technically anything could be overridden, for as long as the entry itself is not deleted and no new entries are added, even reference name.

Keep in mind though that this is all done at compile-time. You might be able to change item name, and randomize it as necessary, but once the module is compiled, the results of that randomization will persist (at least until next compilation). So any player who runs the mod will always see the same item name (even if not necessarily the one that it had in the original source files).

...as well as having an item act differently (different trigger called, possibly changing models if that would be possible) depending on which team the player wielding it is on.

Especially overriding the model could be really useful, I can see people using it for damaging equipment.

All these things have to be done at run-time, so it's a different story. You might be able to add extra models to an item at compile-time, tying them to specific imod-values, but actually changing them will require some run-time code (most likely a mission template trigger, or it's combination with item trigger).

Well, technically this means that your plugin would have to add both extra models for an item, and a mission template trigger to handle them.

 

[Arch3r]

Ah woops, I thought it would allow for runtime overriding, my bad. I suppose it's good for disabling features just for a mod release though. Then you can develop the mod and simply disable some parts with an override if you are doing weekly releases.