New Module System Compiler Development

Users who are viewing this thread

Total: 2 (members: 0, guests: 2)
Status
Not open for further replies.
Lav said:
Okay, a brief version of feature list. I'll expand it and add to the starting post later, this is just the essence.

  • Much faster (6 seconds vs 24 seconds for vanilla Native compiler).
  • Much more graceful and (generally) more informative syntax error reporting.
.
Click to expand...

My eyes is good ? Or my head is gave 404 brain not found. :smile:

And also ''OOP-style syntax support.'' What means ? For example,complete's the undefined code ?
 
Ra'Jiska said:
Sorry, I might not have been clear enough.
What I meant by obfuscation / 'anti reverse-engineering', I was refering to the code generated, not the compiler.
By doing so, it'd make it harder for 'bad people' to steal mod sources by 'decompiling' it.
Click to expand...
You have been clear enough, and that's exactly what I meant. I'm much more likely to start with the idea of a de-obfuscating decompiler than the obfuscating compiler. :smile:

HyperCharge said:
My eyes is good ? Or my head is gave 404 brain not found. :smile:

And also ''OOP-style syntax support.'' What means ? For example,complete's the undefined code ?
Click to expand...
Not sure about your eyes, but the syntax is not much for now. Generally, it means you can write trp.player instead of "trp_player". Essentially, you don't have to type the quotes around everything (which are one of the few things to really annoy me in Warband modding), but there are some additional benefits to this. Namely, with this syntax you can write something like:
Code: 
(str_store_string, s1, s.center_relation_00 + min(l.center_relation / 10, 9))
And the compiler will auto-generate the code to divide the visited center's relation by 10, bring it into 0..9 range and add the resulting offset to the reference of the first string of the relation strings block.

Or look how the module_constants.py defines num_trade_goods constant. Namely:
Code: 
num_trade_goods = itm_siege_supply - itm_spice
To define the constant, module_constants.py has to import ID_items.py file which contains the item references as they were after previous compilation. However what happens when you insert a new item into that range and compile? References in the ID file are obsolete until after your compilation succeeds which means that your first compilation will actually produce incorrect code. Which means you generally have to run the compiler twice to get correctly compiled code.

With object-oriented references, their actual values from the current compilation pass are always used, meaning that a similar-looking expression...
Code: 
num_trade_goods = itm.siege_supply - itm.spice
...will always produce correct code on the first run.

For similar reasons, using the vanilla compiler you cannot add a new map icon to a module and immediately use it for some party or party template. Instead, you have to add a new map icon, compile your module (generating updated ID_map_icons.py file), use it for your parties and then compile your module again. This is actually a regularly recurring question in the Q&A thread (and Forge in general).

And frankly, waiting for 20-25 seconds for the compiler to finish compiling some minor change you made gets pretty annoying pretty fast. Doing that twice to be sure everything is correct is annoying four times that much.

There's more to that, but these are the basics. :smile:
 
Wow!  :shock:

I have an another question :smile: for example,in vanilla,we making and adding code.then we compiling,and no errors in cmd screen.but,when we opened the game,and gives me long of ''error in opcode script_create_kingdom_hero_party'' this is an example :smile: okay.but when we added to module,and when we compiled,we had no errors.but in game,gave an millions of errors with red color.why it not giving in compile screen and why it gives on the game ? Also,this greatest thing fixing the devil red colors of errors ??
 
Compiler only ensures your code doesn't contain syntax errors (like attempt to use an unknown operation, or trying to assign a value to a number - though in Warband scripting, the latter actually works). Making sure that your code actually makes sense and doesn't try to obtain party template of party #-1 (one of fairly common opcode error-producing mistakes) falls entirely to you no matter what compiler you use.
 
HyperCharge said:
Wow!  :shock:

I have an another question :smile: for example,in vanilla,we making and adding code.then we compiling,and no errors in cmd screen.but,when we opened the game,and gives me long of ''error in opcode script_create_kingdom_hero_party'' this is an example :smile: okay.but when we added to module,and when we compiled,we had no errors.but in game,gave an millions of errors with red color.why it not giving in compile screen and why it gives on the game ? Also,this greatest thing fixing the devil red colors of errors ??
Click to expand...
Compilers just check for syntax and statement errors; they can't account for every possibility that could actually come into play in-game. Sometimes, a piece of code seems to work perfectly until it breaks down under some rare and random circumstances.
 
Lor Dric said:
will this work with modmerger? I have over 200 files in my new mods module system could this handle that?
Click to expand...
That's an interesting question actually. I've never worked with ModMerger, but from what I see, it seems to do something similar to my plugin system, except it merges the code at import time. This means that my compiler should be compatible with modmerger but I don't think you'll be able to safely use more advanced features of my compiler. Essentially what you'll get will be: faster compilation, ability to compile module_item_modifiers, module_user_hints and module_ui_strings as part of your module, and error reporting (of reduced quality because the compiler won't be able to detect what ModMerger plugin caused the error but at least you won't get 5+ pages of error messages because of a single typo). Other features actually might work, but I cannot guarantee that at the moment.

P. S. Actually I think you should be able to use my version of plugins together with ModMerger's plugins without any issues - at least simple ones that don't require code injection. Though the mess that would create for module support... :twisted:
 
Okay, I did some testing with ModMerger and the situation seems much better than I thought. :smile:

You won't be able to use plugins with code injection in combination with ModMerger - ModMerger's code is not aware of injection markers and the compilation will fail with error. The only exception are the injection points defined in plugins themselves - these will be perfectly fine. Unfortunately, I have yet to make at least one such plugin. :smile:

Without making any modifications to module files, compiler will work without issues, but will lack advanced features - same as it would with any other unadapted module. ModMerger changes nothing in this respect, except the aforementioned conflict with more advanced plugins.

Module files can be adapted by adding:
Code: 
from compiler import *
...at the beginning of the file, preferably after all other import directives. This will enable compiler's full functionality. Plugins using code injection will still conflict with ModMerger but everything else should work without issues.

Note that adding the from compiler import * directive before other imports may disable or override some compiler's troop-, item- and skill-related features (skill constants that compiler declares using actual values from module_skills.py being overwritten by predefined ones in header_skills.py, or compiler's hacks for improved handling of troop/item parameters being cancelled, that kind of stuff). It will not break the compiler, just strip some features off fully or partially.
 
Technically, your plugin can import base mission templates module, for as long as you redefine the templates array. That will give you all trigger definitions from there.
 
is there a way to add a toll to make it easier to find where is the problem with your try_begins/ends? sth similar to notepad++'s    cntrl + b  where if you use it near a [ it will lead to the ] it is connected and vice versa.  cause I'm trying to fix script_game_receive_network_message and it's such a pain...
 
Ikaguia said:
is there a way to add a toll to make it easier to find where is the problem with your try_begins/ends? sth similar to notepad++'s    cntrl + b  where if you use it near a [ it will lead to the ] it is connected and vice versa.  cause I'm trying to fix script_game_receive_network_message and it's such a pain...
Click to expand...
I'm planning to add the information on how many try_end's are missing, or how many are extra, especially since that information is already available for compiler, but there's little else that can be done.
 
Devlog for next release:
  • Added information on how many try_end's are missing/extra in code blocks to error output.
  • Added generation of modifier-related constants (imod_* and imodbit_*) derived from module files for backwards-compatibility.
  • Implemented the framework for direct access to entity properties. Only items properties are currently implemented as the feature is still in testing phase. So it is now possible to write things like (assign, g.damage, itm.broadsword.thrust_damage). Retrieving those properties during run-time will obviously still require ms extension plugin, using slots and scripts.
  • A few other minor modifications I forgot the specifics of.
Another feature I'm currently considering is syntax extension plugins. So, for example, a plugin could override (set_fixed_point_multiplier, <value>) operation so it would be automatically converted to (set_fixed_point_multiplier, <value>),(assign, "$_fp_multiplier", <value>) and define an extra (get_fixed_point_multiplier, <destination>) operation which would convert into a single (assign, <destination>, "$_fp_multiplier"). This conversion would then happen automatically and completely transparently throughout the entire code (including other plugins).

There are limits to this feature - it shouldn't be possible to extend can-fail operations as they can be part of this_or_next sequences, and any new can-fail operations should always consist of a single operation, but otherwise seems to be a potentially useful tool.
 
Almost ready for official release.

I'm actually using this compiler extensively for Native Expansion mod for about two months by now.

Latest changes:
  • Compiler now generates knows_<skill>_<level> constants correctly for all possible skill levels, not just first 10.
  • Made a number of changes to compiler robustness and to improve messages readability.
  • It is now possible to extend Module System syntax by adding your own operations (for as long as those operations are converted to a sequence of legal vanilla opcodes).
To illustrate the last point, the following code actually works without a hitch:

Code: 
from compiler import *
register_plugin(__name__)

slot_party_struct_type   = 0
slot_party_struct_count  = 1
slot_party_struct_start  = 2
slot_party_struct_end    = 3
slot_party_struct_items  = 4 # Actual items are stored starting from this slot

struct_type_set = 0
struct_type_list = 1
struct_type_ordered_list = 2
struct_type_stack = 3
struct_type_queue = 4

# This will make these values accessible to the rest of module system
export_plugin_globals(
    struct_type_set          = struct_type_set,
    struct_type_list         = struct_type_list,
    struct_type_ordered_list = struct_type_ordered_list,
    struct_type_stack        = struct_type_stack,
    struct_type_queue        = struct_type_queue,
)

map_icons = [
  ("data_structure", mcn_no_shadow, "battle_track", 1.0, 0),
]

party_templates = [
    ("data_structure", "{!}", icon.data_structure|pf_hide_defenders|pf_is_static|pf_no_label|pf_disabled, 0, fac.no_faction, 0, []),
]

def struct_create(destination, struct_type = struct_type_set):
    return [
        (assign, l._cached_, reg0),
        (spawn_around_party, p.temp_party, pt.data_structure),
        (party_set_slot, reg0, slot_party_struct_type, struct_type),
        (party_set_slot, reg0, slot_party_struct_count, 0),
        (party_set_slot, reg0, slot_party_struct_start, -1),
        (party_set_slot, reg0, slot_party_struct_end, -1),
        (assign, destination, reg0),
        (assign, reg0, l._cached_),
    ]

def struct_destroy(struct_id):
    return [
        (remove_party, struct_id),
    ]

def struct_set_name(struct_id, new_name):
    return [
        (party_set_name, struct_id, new_name),
    ]

def str_store_struct_name(destination, struct_id):
    return [
        (str_store_party_name, destination, struct_id),
    ]

extend_syntax(struct_create)
extend_syntax(struct_destroy)
extend_syntax(struct_set_name)
extend_syntax(str_store_struct_name)

# TEST CODE

simple_triggers = [
    (0, [
        (key_clicked, key_k),
        (map_free),
        (struct_create, ":tmp"),
        (struct_set_name, ":tmp", "@Test Struct Name"),
        (str_store_struct_name, s1, ":tmp"),
        (struct_destroy, ":tmp"),
        (display_message, "@Test successful. Struct name was `{s1}`."),
    ]),
]
Of course, the idea is that these operations, despite being defined in a plugin, become available to the rest of the Module System without any extra effort. Essentially, this is the "syntax extension plugin".
 
  • Lav said:
    • It is now possible to extend Module System syntax by adding your own operations (for as long as those operations are converted to a sequence of legal vanilla opcodes).
    Click to expand...

    What the.... :shock:

    you meaning "i can add my own operations" ?

    Thats impressive!
 
If I understand this correctly,

(try_for_range, ":center_no", centers_begin, centers_end),
(party_slot_eq, ":center_no", slot_party_type, spt_village),

could be replaced with:

(try_for_village)

That seems to be a great way of simplifying the code.
 
Fire_and_Blood said:
If I understand this correctly,

(try_for_range, ":center_no", centers_begin, centers_end),
(party_slot_eq, ":center_no", slot_party_type, spt_village),

could be replaced with:

(try_for_village)

That seems to be a great way of simplifying the code.
Click to expand...
Sure, why not. This is essentially a macro substitution mechanism (or a more powerful code generating mechanism for someone who knows a bit of Python). So you can put pretty much anything there.

Of course the original idea was for experienced modders to create syntax-extending libraries, not for everyone and their dog to create mutually-incompatible syntax tweaks. But, well, the mechanism is there and it's probably going to get abused anyway... :wink:
 
Lav said:
Almost ready for official release.

I'm actually using this compiler extensively for Native Expansion mod for about two months by now.

Latest changes:
  • Compiler now generates knows_<skill>_<level> constants correctly for all possible skill levels, not just first 10.
  • Made a number of changes to compiler robustness and to improve messages readability.
  • It is now possible to extend Module System syntax by adding your own operations (for as long as those operations are converted to a sequence of legal vanilla opcodes).
To illustrate the last point, the following code actually works without a hitch:

Code: 
from compiler import *
register_plugin(__name__)

slot_party_struct_type   = 0
slot_party_struct_count  = 1
slot_party_struct_start  = 2
slot_party_struct_end    = 3
slot_party_struct_items  = 4 # Actual items are stored starting from this slot

struct_type_set = 0
struct_type_list = 1
struct_type_ordered_list = 2
struct_type_stack = 3
struct_type_queue = 4

# This will make these values accessible to the rest of module system
export_plugin_globals(
    struct_type_set          = struct_type_set,
    struct_type_list         = struct_type_list,
    struct_type_ordered_list = struct_type_ordered_list,
    struct_type_stack        = struct_type_stack,
    struct_type_queue        = struct_type_queue,
)

map_icons = [
  ("data_structure", mcn_no_shadow, "battle_track", 1.0, 0),
]

party_templates = [
    ("data_structure", "{!}", icon.data_structure|pf_hide_defenders|pf_is_static|pf_no_label|pf_disabled, 0, fac.no_faction, 0, []),
]

def struct_create(destination, struct_type = struct_type_set):
    return [
        (assign, l._cached_, reg0),
        (spawn_around_party, p.temp_party, pt.data_structure),
        (party_set_slot, reg0, slot_party_struct_type, struct_type),
        (party_set_slot, reg0, slot_party_struct_count, 0),
        (party_set_slot, reg0, slot_party_struct_start, -1),
        (party_set_slot, reg0, slot_party_struct_end, -1),
        (assign, destination, reg0),
        (assign, reg0, l._cached_),
    ]

def struct_destroy(struct_id):
    return [
        (remove_party, struct_id),
    ]

def struct_set_name(struct_id, new_name):
    return [
        (party_set_name, struct_id, new_name),
    ]

def str_store_struct_name(destination, struct_id):
    return [
        (str_store_party_name, destination, struct_id),
    ]

extend_syntax(struct_create)
extend_syntax(struct_destroy)
extend_syntax(struct_set_name)
extend_syntax(str_store_struct_name)

# TEST CODE

simple_triggers = [
    (0, [
        (key_clicked, key_k),
        (map_free),
        (struct_create, ":tmp"),
        (struct_set_name, ":tmp", "@Test Struct Name"),
        (str_store_struct_name, s1, ":tmp"),
        (struct_destroy, ":tmp"),
        (display_message, "@Test successful. Struct name was `{s1}`."),
    ]),
]
Of course, the idea is that these operations, despite being defined in a plugin, become available to the rest of the Module System without any extra effort. Essentially, this is the "syntax extension plugin".
Click to expand...

You have won me over. Downloading.
 
Status
Not open for further replies.

Similar threads

Persistent WRECK error when compiling module_troops.py
Replies
1
Views
83
hounskulll
hounskulll
Need halp freelancer modmerger!!!
Replies
2
Views
131
Idrank2muchblood
[Native 1.174] Very easy request. How to force enemy AI to always charge immediately?
Replies
2
Views
222
aleeque
Back
Top Bottom