New Module System Compiler Development

A small status update.

ID files are properly generated now. Also, I finalized the vision of savegame compatibility feature, but it's quite complex and won't be properly implemented for at least a couple more versions.

Plugins overriding/overwriting values for existing data entries will probably be done with a filter/callback scripts. These will be called after data injections are done and identifiers allocated (so all entries are valid and no entries can be deleted/inserted at this point), but before actual compilation and script code injections. Hence it will be possible for a plugin to create a filter which will not override anything, but simply collect information about all game entries of a certain type, and generate a script code injection to initialize these entities during the game, or store some extra data about them etc.

Also still need to generate warnings when some data and/or code injections provided by the plugin are not actually used. Generally this means that injection point was lost, and may result in non-functioning plugin, so this warrants a warning to the modder.

Finally, introducing item modifiers as module entity warrants some modifications to the code, as the compiler no longer can assume all module files will be present for compilation. Otherwise, it's pretty straightforward.

Once all this is done, version 0.4 beta can be safely released, probably as two versions: one to be used with vanilla Module System (copy&compile), and another as a fully adapted Module System with all identifier and variable names translated to new syntax, ID-file being purely optional etc.

Beta Release.

Not all features planned for this release are implemented, but ID files are generated so compiler is now backwards-compatible with vanilla compiler script.

Plugins may now define their own preprocess_entities script. It will be called at a point where all entity-level injections are already done and all identifiers are already calculated. This means that preprocess script must not add/delete/insert/replace module entities, but it's still perfectly possible to access their data and make some script-level injections.

Version 0.4 Beta on Google Drive.

As not all planned features are implemented, I'm treating this as a preliminary release. Thus, I'm releasing it without any extras. Namely, this release lacks any plugins, nor does it contain a fully adapted Module System.

Usage instructions are as always: extract archive contents into module folder and run compile.py script. Voila.

I think it's easy to make compiler able to create new variables while compiling. And I know you can do it

I really have no idea what python codes look like, this is a completely made-up code example But results should be the same with results of those:


So both should make variable_1 = 1, variable_2 = 2, variable_3 = 3, variable_4 = 4, variable_5 = 5.

Erm... I don't see how that would actually be useful. Can you provide a real-life example?

I need this for 2 small jobs for now

(assign, ":value", 1),
(assign, ":y", 1000),

(store_skill_level, ":level", "skl_inventory_management", "trp_player"),
(assign, ":base_inventory_slot_count", 30),
(val_mul, ":level", 2),
(store_add, ":cur_inventory_slot_count", ":base_inventory_slot_count", ":level"),

(variable.loop = "$g_inventory_slot_(x)", 1, ":cur_inventory_slot_count"),
(val_add, ":value", 1),
(create_mesh_overlay, "$g_inventory_slot_(x)", "mesh_inv_slot"),
(position_set_x, pos1, 0),
(val_sub, ":y", 100),
(position_set_y, pos1, ":y"),
(overlay_set_position, "$g_inventory_slot_(x)", pos1),
(end.variable.loop),

Click to expand...

[So we can check every object we created by it's name.]

(variable.loop = "$g_presentation_obj_admin_panel_(x)", 1, 25),
(assign, "$g_presentation_obj_admin_panel_(x)", -1),
(end.variable.loop),

Click to expand...

[Or define too much variables with a short script]

Those can be handled with different methods but this way is fastest

Probably fastest, but unfortunately impossible.

You want to mix compiler-time and run-time iteration. This will not work, like, at all.

And you cannot iterate through global variables anyway, even if they are created in sequence.

Use slots to store your overlays. That's what I did in Companions Overseer, and guess what? I did it this way for a reason.

Lav said:

Use slots to store your overlays.

Click to expand...

Yeah "different methods" meant to this. Allright then. That was just a suggestion, which would be perfect

Development Sneak-Peek.

1. Implemented plugin requirements - it is now possible for plugins to only be functional if a certain other plugin (or plugins) is loaded as well.
2. Implemented module_item_modifiers as a separate optional module file, exporting to Data/item_modifiers.txt if present.
3. Implemented module_ui_strings as a separate optional module file, exporting to Languages/en/ui.csv if present.
4. Implemented module_user_hints as a separate optional module file, exporting to Languages/en/hints.csv if present.
5. Developed a preliminary version of plugin_ms_extension which is expected to contain most of Module System extensions (accessible attribute/skill/proficiency/item_modifier names extracted automatically from module_ui_strings, module_skills and module_item_modifiers; helper scripts to gain access to various item/troop/scene/etc information that's normally unavailable for Module System code, et cetera, ad infinitum). At the moment only string collection is implemented but plugin is still in early development.
6. Fixed a bug in reference name resolution code.

All things considered, release 0.5 is going to be everything that release 0.4 was supposed to be with a little sugar on top. It's also going to include a fully adapted Native Module System and a set of various plugins (plugin_ms_extension described above, plugin_lists with helper functions for list-based data storage, plugin_presentations_ui with helper functions for easier presentations generation - still in early development though; and plugin_mining with plugin_companions_overseer as an extra bonus).

Date for version 0.5 release is still not clear, but I'm already on the finishing line.

Apparently I got on a rush.

Version 0.5 Beta available for download.

Look previous message for contents or follow the URL for a more detailed description.

Bugfix release: v.0.5.1.

Fixed an exotic bug in preprocess_entities() function handling for plugins. Found it when my heraldry support plugin, which was supposed to add tableau support to banner icons suddenly added tableau support to most map icons in the module, resulting in towns, castles, bridges etc painted in heraldic colors.

Should not affect anyone as I doubt someone already exploited the possibilities provided by the plugin system to that extent.

On another note, it seems there are issues with troop skills, so don't use this for live project until it's fixed.

UPDATE: False alarm. I put the skills into Excel spreadsheet to generate the troop tuples, copy-pasted them into module_troops, compiled and saw troops with no skills. But the bug was not in the compiler, but in my troop tuple generator.

Version 0.5.2 Beta available for download.

• It is now possible to add items with modifiers to troop inventory. Just type [(itm.bread, imod.tasty), ...] instead of [itm.bread, ...]. Both variants can be used interchangeably. This means lords and kings now can get lordly horses and masterwork swords.
• Made references case-insensitive (need feedback on this one, might be better to preserve case sensitivity).
• Fixes a couple of bugs causing abnormal compiler termination instead of gracefully reporting syntax errors in the module.

Version 0.5.3 Release Candidate available for download.

• Bugfix: code injection resulted in duplicated and/or lost code when using more than one plugin.
• Bugfix: compiler's alternative upgrade() definitions for troops were not backwards-compatible with vanilla Native.
• Plugin bugfix: fixed plugin_companions_overseer issues when using some of the more obscure functionality of the plugin.
• Modified plugin_companions_overseer plugin to display item's requirements for both player and currently selected hero (if there's one).
• Added "script_get_item_prerequisites" script to module_ms_extension.
• Extended the range of available registers (registers up to reg99, s99 and pos99 are now available).
• Implemented automatic generation of skill-related constants (skl_%skill%) and (knows_%skill%_%level%) to support vanilla-style troop declarations in the fully converted module system (not converted module system will have it's own constants declared in header_skills.py anyway so this is a non-issue there).
• Various lesser improvements and bugfixes.

Performance comparison.

Vanilla Module System compiler, compiling Native 1.158 sources without any modifications: 23.15 seconds (23.05 to 23.35 seconds in different tests).
Version 0.5.3 RC1 compiler (quick install variant), compiling the same (unadapted) sources: 5.95 seconds (5.90 to 6.00 seconds in different tests).
Version 0.5.3 RC1 compiler compiling the fully adapted sources: 5.85 seconds (very stable results in several different tests).

Excellent work, keep it up!

Also, I believe you have a typo in script.pui_container: passing x and y to [font=consolas,courier]overlay_set_size[/font] instead of [font=consolas,courier]overlay_set_position[/font].

Seems so, thanks for the hint. Unfortunately for this iteration I haven't made any progress with this plugin, so overseer plugin is still using it's own helper functions (which is what I'm basing Presentations UI plugin on), but it's one of the goals for compiler's actual release (along with Lists plugin which is going to be expanded into Data Structures, and of course MS Extension).

sorry for my ignorance but,what it does ? or what is different from standart ? also should i use that ?

Hmm, perhaps it's time to make a proper feature list.

Okay, a brief version of feature list. I'll expand it and add to the starting post later, this is just the essence.

• Much faster (6 seconds vs 24 seconds for vanilla Native compiler).
• Much more graceful and (generally) more informative syntax error reporting.
• OOP-style syntax support.
• ID_* files are optional - single-pass compilation no matter what changes are made to the module.
• Arithmetic calculations using object references at compile-time.
• Automatic code generation for run-time arithmetic calculations.
• Automatic skills and skill-related values generation - no need to manually fix header_skills in case of any changes to module_skills.
• Item modifiers can be changed as part of module, auto-generating all related values - no need for modder to manually fix header_item_modifiers and Data/item_modifiers.txt.
• User hints can be changed as part of module.
• User interface strings can be changed as part of module.
• Info pages can be referenced and linked properly in the code.
• Native compiler restrictions on code removed:
  • 100 registers of each type (up to reg99, s99 and pos99).
• Native compiler restrictions on items removed:
  • Arbitrary large and small item weights with much higher precision (up to 0.01, though Warband will still round the weights to 0.1 but will track the hundredths correctly).
  • Armor values higher than 255 supported.
  • Arbitrary high HP values supported.
  • Arbitrary high Quantity values supported.
• Native compiler restrictions on troops removed:
  • Troops can have items with modifiers in their inventory without adding them by script.
• Plugin support (this is a huge one actually, deserving it's own feature list ).
• Code injection support for plugins.
• A number of vanilla Module System compiler bugs fixed and/or circumvented.

And of course, it is (or at least should be, unless I messed up somewhere) savegame-compatible with vanilla compiler. Which means you can compile a module using vanilla compiler, start a game, save it, re-compile the module using my compiler and load the game successfully without any issues.

Nice work.
If it can help you, on PW, registers goes up to 127.
Maybe should you also add some obfuscation / anti reverse engineering feature, I'm sure it'd be much appreciated.

Ra'Jiska said:

Nice work.
If it can help you, on PW, registers goes up to 127.

Click to expand...

I know - but string substitution only works up to 99, which is why I chose this number - to prevent less knowledgeable modders from running into problems and coming to me with complaints .

And it's hard to imagine someone who would really need more than 100 registers anyway. Though I'll probably extend position registers up to 127th, that might actually be useful.

Ra'Jiska said:

Maybe should you also add some obfuscation / anti reverse engineering feature, I'm sure it'd be much appreciated.

Click to expand...

Erm... With my personal views, I'm much more likely to write a de-obfuscating decompiler instead. Especially since I do have quite a number of heuristic ideas for one.

Better don't give me any ideas, or the community might start cursing our names forever.

Lav said:

Ra'Jiska said:

Maybe should you also add some obfuscation / anti reverse engineering feature, I'm sure it'd be much appreciated.

Click to expand...

Erm... With my personal views, I'm much more likely to write a de-obfuscating decompiler instead. Especially since I do have quite a number of heuristic ideas for one.

Better don't give me any ideas, or the community might start cursing our names forever.

Click to expand...

Sorry, I might not have been clear enough.
What I meant by obfuscation / 'anti reverse-engineering', I was refering to the code generated, not the compiler.
By doing so, it'd make it harder for 'bad people' to steal mod sources by 'decompiling' it.