[KIT] partitioned Module System source files

Users who are viewing this thread

Just whipped up a simple way to divide up module sources into multiple files within a seperate folder per module_*.py and thought I should share it.
... Probably already exists in some form, but the search function is terrible and it only took a few lines, so here:
Code:
import os
import shutil
from os import path

def parse_directory(dir, maxDepth):
  entries = os.listdir(dir)
  dirs  = [ path.join(dir, d) for d in entries if path.isdir (path.join(dir, d)) ]
  files = [ path.join(dir, f) for f in entries if path.isfile(path.join(dir, f)) and path.splitext(f)[1] == ".py" ]
  for name in files:
    # write from source files
    file = open(name, 'r')
    for line in file:
      # keep the comments out
      if line.find('#') != -1:
        line = line.split('#')[0] + '\n'
      # keep the empty lines out
      if not line.isspace(): # "empty" lines still contain the line separator
        module.write(line)
    file.close()
  if(maxDepth > 0):
    for name in dirs:
      # parse subfolders
      parse_directory(name, maxDepth - 1)

modules = [ m for m in os.listdir(".") if path.isfile(m) and m.startswith("module_") ]
# remove files nonsensical to parse
modules.remove("module_info.py")
modules.remove("module_constants.py") # not to be appended to, as slots et al. need to be conflict free; checking easier in place
for m in modules:
  # fetch directory name for module file: capitalised right side of file name without extension
  dir = path.splitext(m.partition("_")[2])[0].capitalize()
  # if path exists, parse
  if(path.isdir(dir)):
    print("Importing " + dir.lower() + "...") # Python3 compliant
    # save original module_scripts.py
    shutil.copy2(m, m + ".bak")
    # append all source files to module_scripts.py
    module = open(m, 'a')
    parse_directory(dir, 10) # arbitrary limit; may be raised considerably before errors occur
    module.close()
Code:
import os
from os import path

print("Performing cleanup operations...")
files = [ f for f in os.listdir(".") if path.isfile(f) and path.splitext(f)[1] == ".bak" ]
for name in files:
  orig = path.splitext(name)[0]
  # restore original source files
  os.remove(orig)
  os.rename(name, orig)
Also Python3 compliant, by the way. I actually developed them with a Python3.3 executable.
Add before process_init.py:
Code:
<Python path>\python.exe process_imports.py
Add after process_postfx.py:
Code:
<Python path>\python.exe process_cleanup.py
Use "<array>.extend([[...]])" to append to arrays (such as array "scripts" in "module_scripts.py").
When defining local constants, do not use indentation! These are interpreted as Python code and indentations are semantic information for Python. If you want to write functions, I assume you know your trade.

Files are fetched from the directory associated with the main file ("Animations" for "module_animations.py", "Scripts" for "module_scripts.py" etc.) and any subdirectories down to a depth of 10.
Any name will do. As long as the file has the correct suffix ".py", it will be fetched.
Code:
scripts.extend([
  #script_set_adjacency_between_centers
  # INPUT : arg0 = center1, arg1 = center2, arg2 = new adjacency value
  # OUTPUT: none
  ("set_adjacency_between_centers",[
    (store_script_param_1, ":center1"),
    (store_script_param_2, ":center2"),
    (store_script_param, ":adjacent", 3),

    (party_get_slot, ":adjacencies", ":center1", slot_center_adjacencies),
    (party_count_members_of_type, ":adjacencyValue", ":adjacencies", ":center2"), # hack: put partyId as troopId
    (party_remove_members, ":adjacencies", ":center2", ":adjacencyValue"),
    (party_force_add_members, ":adjacencies", ":center2", ":adjacent"),

    (party_get_slot, ":adjacencies", ":center2", slot_center_adjacencies),
    (party_count_members_of_type, ":adjacencyValue", ":adjacencies", ":center1"),
    (party_remove_members, ":adjacencies", ":center1", ":adjacencyValue"),
    (party_force_add_members, ":adjacencies", ":center1", ":adjacent"),
  ]),
])
 
A heads up to anyone who might have tried this out before 23.02.2013, 11.00 pm GMT+1:
There was a pasting error in the import script, which would lead to only "module_scripts.py" receiving its backup. This would of course cause every other extended file to accumulate extensions.

Current version is fine, though. A bit optimised, too.
 
If you want to split your files into more sources (a common way to add many/long scripts or mission templates and keep them apart from the Native stuff), you could simply split their arrays into two and put them in different files, then have one file import the other and concatenate the arrays.
 
If Python imports worked like C includes... as it is, I cannot import from relative folders without a messy workaround.
At least that is what I have gathered. I have not much experience with Python.
MadVader said:
[Y]ou could simply split their arrays into two and put them in different files, then have one file import the other and concatenate the arrays.
It has occurred to me... which is why I did - exactly that. Crazy, eh?
 
Download swysdk and see how I did. This and most part of the tricks were originally though and implemented in the swconquest msys.

Just by altering the batch script and adding the various folders to the (local instance of)
Code:
%PATH%
.
Way easier than your wall of scripts.

Here's the repo with its changeset, uses Hg.
Code:
https://bitbucket.org/Swyter/swysdk/commits/all

Specific change in the
Code:
build_module.bat
script:
Code:
https://bitbucket.org/Swyter/swysdk/commits/ad4577fc400ec70dfbcc07697c6940ce8f4f56ce#Lbuild_module.batT16

Small modification in the process files to output on the
Code:
/ID
dir instead of the base:
Code:
https://bitbucket.org/Swyter/swysdk/commits/ad4577fc400ec70dfbcc07697c6940ce8f4f56ce#LProcess/process_factions.pyT61

And that's pretty much it. swysdk stores a lot more more goodies, of course.
Feel free to find them all and use them in your own benefit.
 
So there really is no manual...
As far as I can see, pathes are included by explicit notion in build_module.bat.
I therefore have to implement the same system I used here in Windows batch script instead of Python, build my PYTHONPATH on runtime, then edit the modules in question to include import-statements for all files found. As I cannot (at least: know not how to) append to a file anywhere but at the end, for that I have to rewrite every module file on runtime (which I would rather do in Python or even C - but then why not use the existing code that already is done).
... Really, that seems more trouble than it is worth. I do not mind to invent different wheels, but the same wheel multiple times!?

- Feel free to correct me if I misjudge things. I really hope I do, in fact.
 
Zsar said:
So there really is no manual...
As far as I can see, pathes are included by explicit notion in build_module.bat.
I therefore have to implement the same system I used here in Windows batch script instead of Python, build my PYTHONPATH on runtime, then edit the modules in question to include import-statements for all files found. As I cannot (at least: know not how to) append to a file anywhere but at the end, for that I have to rewrite every module file on runtime (which I would rather do in Python or even C - but then why not use the existing code that already is done).
... Really, that seems more trouble than it is worth. I do not mind to invent different wheels, but the same wheel multiple times!?

- Feel free to correct me if I misjudge things. I really hope I do, in fact.
Nonono. No import statements, or anything like that. The single Python file is a separate wrapper used to run the processors in the same instance, hence skipping the initialization all-together plus sharing imports, which means: faster. But that's another story.

---

The magic works by creating the following structure:
Code:
/msys-folder
├───Headers -> header_*.py
├───IDs     -> id_*.py
└───Process -> process_*.py
module_*.py
build_module.bat


The trick is to set all these directories comma-separated in a global variable called
Code:
%PYTHONPATH%
, so Python knows where it has to search for
Code:
.py
files. With this simple override we are able to put them anywhere we want.

As seen below, in my case I also bundle the Python interpreter, so I added in the
Code:
%PATH%
the directory where
Code:
python.exe
is located and Windows will crawl it in case wasn't found in any other place. all these variable assignments are temporal, and only for this sole batch script. Yet again, this paragraph is optional, and of course, setting it via the properties panel on the system settings will have the same result, as shown in the official setup tutorial.

Code:
:: @> hi there! I'm pretty honored to see you here,
::    please feel free to copy anything. -Greetings from Swyter
::   -------------------------------------------------------------

SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
MODE CON: COLS=40 LINES=40

:init
@echo off
cls && color 71 && title [ ] swysdk -- building

:: this is to support paths with spaces and strange characters
set CD="!CD!"

:: setup our python and specify what folders are included in the search path for scripts
set PATH=!CD:~1,-1!\Python
set PYTHONPATH=%PYTHONPATH%;!CD:~1,-1!\IDs;!CD:~1,-1!\Headers;!CD:~1,-1!\Process;!CD:~1,-1!

The second part is optional and consists in adding a -B param to the Python executable, so it doesn't generates compiled
Code:
.pyc
files which are deleted later anyway.

As now our process files are in a separate subdirectory we will update it accordingly.

Code:
:: 
:: the -B param overides the pyc/pyo bytecode generation, so there's no need for deleting them later :)
python -B Process/process_init.py
python -B Process/process_global_variables.py
python -B Process/process_strings.py
python -B Process/process_skills.py
python -B Process/process_music.py
python -B Process/process_animations.py
python -B Process/process_meshes.py
python -B Process/process_sounds.py
python -B Process/process_skins.py
python -B Process/process_map_icons.py
python -B Process/process_factions.py
python -B Process/process_items.py
python -B Process/process_scenes.py
python -B Process/process_troops.py
python -B Process/process_particle_sys.py
python -B Process/process_scene_props.py
python -B Process/process_tableau_materials.py
python -B Process/process_presentations.py
python -B Process/process_party_tmps.py
python -B Process/process_parties.py
python -B Process/process_quests.py
python -B Process/process_info_pages.py
python -B Process/process_scripts.py
python -B Process/process_mission_tmps.py
python -B Process/process_game_menus.py
python -B Process/process_simple_triggers.py
python -B Process/process_dialogs.py
python -B Process/process_global_variables_unused.py
python -B Process/process_postfx.py
title [X] swysdk -- finished
echo ______________________________
echo Script processing has ended.
echo Press any key to restart. . .
pause>nul
goto :init


Difficult? convoluted? Seriously? Tidy and easy, I'd say.
 
Wait, wait, I think I must elaborate the requirement I have, as I find no hint towards accomplishing it in this explanation:

Say, I have a folder Headers, the headers are in there. Fine.
Now, I want to create a folder Scripts, wherein I store scripts, to be appended to module_scripts.py .
Now, I want to create subfolders Graph, Luck, Adjacencies, wherein I store scripts that might reference each other.
Example: Adjacencies uses Graph for graph-based algorithms such as finding a path. It might also invoke a script from Luck to discern, whether among fairly equal pathes a lord pick the best, the worst or one inbetween.

My kit performs depth-first traversal through the directory structure and appends it all to module_scripts.py before processing is started.
In other words, to drop a file into any subdirectory of Scripts suffices for its inclusion and its position relative to any other subfolder is irrelevant.
Simple and clear.

How do I do exactly that with swysdk?
  • Declare folder F matched to one module file.
  • Have Drag&Drop support for any subfolder of F.
  • Have full automatic access between arbitrary subfolders of arbitrary depth from F.

If I just extended the PYTHONPATH with F, would that suffice?
I think not: None of the code you posted concatenates these source files in any way. None of the code you posted contains a loop with dynamic range to process an arbitrary number of files sequentially.
If it does work that way, it does so by a mechanism that completely eludes me.
 
Zsar said:
Wait, wait, I think I must elaborate the requirement I have, as I find no hint towards accomplishing it in this explanation:

Say, I have a folder Headers, the headers are in there. Fine.
Now, I want to create a folder Scripts, wherein I store scripts, to be appended to module_scripts.py .
Now, I want to create subfolders Graph, Luck, Adjacencies, wherein I store scripts that might reference each other.
Example: Adjacencies uses Graph for graph-based algorithms such as finding a path. It might also invoke a script from Luck to discern, whether among fairly equal pathes a lord pick the best, the worst or one inbetween.

My kit performs depth-first traversal through the directory structure and appends it all to module_scripts.py before processing is started.
In other words, to drop a file into any subdirectory of Scripts suffices for its inclusion and its position relative to any other subfolder is irrelevant.
Simple and clear.

How do I do exactly that with swysdk?
  • Declare folder F matched to one module file.
  • Have Drag&Drop support for any subfolder of F.
  • Have full automatic access between arbitrary subfolders of arbitrary depth from F.

If I just extended the PYTHONPATH with F, would that suffice?
I think not.

You would have to import (append?) them at the bottom of
Code:
module_scripts.py
like MadVader said.
Python will load the file from anywhere, just take care of adding it to
Code:
PYTHONPATH
.

That's it.
 
So, not worst case: I can still append at the end. Still have to use my kit, just to create import statements instead of copying the whole file content.
Hé, that does make the world a better place.

I will have at that. Mayhap today even. See how well it can work.
... You have not, by chance, hid away a collision detection for constants, slot definitions in particular? That would allow to savely partition module_constants.py, which in turn were a boon.
 
Zsar said:
So, not worst case: I can still append at the end. Still have to use my kit, just to create import statements instead of copying the whole file content.
Hé, that does make the world a better place.

I will have at that. Mayhap today even. See how well it can work.
... You have not, by chance, hid away a collision detection for constants, slot definitions in particular? That would allow to savely partition module_constants.py, which in turn were a boon.

No, I haven't. Luckily, constants are something you don't use or change normally.
If you want to have it more tidy, then why not just define them at the top of the single script file in which you are going to use them? They are just numbers, you know, KISS.
 
Well, overlapping slot definitions just might be a silent, hard to spot problem.
Code:
troop_slot_home = 2
troop_slot_liege  = 2
... Sure, will compile, no problem.
- Oh, then we save a party in troop_slot_home and a troop in troop_slot_liege and get arbitrary results in our code? Tsk tsk tsk.

These have to be collision free by parent type. Without automatic detection, to keep them all in one place is the only way to ensure this.

Now, slot definitions and slot content definitions should be in the same block for ease of reference. Slot content definitions are quite harmless and may be defined locally as you suggest - but then the slot definition should be with them... I need not continue, do I?

I constantly (hè) employ named constants. A good way to increase maintainability.
In my (yet unfinished) adjacency library, a node is stored in a center's party_slot.
Here g_player_luck turned into a troop_slot for the purpose of generalising this value to all unique characters.
In this script for the original M&B I have used an item_slot to store the factions whose traders should have access to it.

I have on average (and indeed: exactly) required one additional slot per project.

addendum:
Oh wait, I just recalled something - to import a file, it must be a valid module, so I had to carry back the necessary imports to the files-to-be-imported, had I not? I recall this issue leading me to the copy-approach in the first place.

Concise:
Code:
Scripts.append([
  ("example_script",[
    (store_val, ":bla", 0, 1),
  ])
])
I would earn on import into module_scripts a "NameError: name 'store_val' not defined" unless I imported header_scripts therein. Same for all other sources.
 
I don't wan't to bring you down, but this is adding complexity unnecessarily, just for the sake of it.
If I have learned something from programming is that the less code, the better.

I know what you want to do, and something like Modmerger is for you.
Instead of splitting files, I add bookmarks in [abbr=Notepad++]npp[/abbr].

For swconquest, which is huge, we count with just an additional file,
Code:
module_mission_templates_common.py
. It stores the plug-triggers, so mission_templates just stores the templates. And for the rest, is pretty maintainable. So, forget the meta-coding thing. It works.
 
cmpxchg8b said:
Swyter said:
If I have learned something from programming is that the less code, the better.
Is it? I guess you could remove the constants and just use numbers and have less code, but I'm not sure that would be better.
Very funny. You probably like this one better:
Antoine de Saint-Exupery said:
Perfection is achieved, not when there is nothing left to add, but when there is nothing left to remove.
 
If it's about the slot definition, even on native we find that they jump over some numbers, but then not all those  numbers are safe to use to define new slots. Some are used by scripts by range.

Sometime it's safe to reuse a number for multiple slots like what I use in my mods:
slot_item_food_bonus = 1
slot_item_book_bonus = 1
It's clear when or what items use it as slot_item_food_bonus and when or what use it as slot_item_book_bonus.

I understand what you want to achieve. You can consider about what I did on enhanced process py. I did exporting all slot definition to ID_constants.py. The file is not used nor imported by any files of module system, but as it's alphabetic  ordered then it will help much to debug some slots using collisions. And as some may think that it costs compiling time, they can switch it off by setting export_constants = 0  at process_commons.py.
 
I still don't get why any of this is necessary.  You can just use includes to bring in anything you need to keep outside of your project; why bother with all of the extra complexity? 

I mean, you may as well just write a Python or grep operation that cuts every single script and builds a separate file in a nested folder (i.e., /scripts, flora_kinds, etc.) and then appends an include statement to a master that's essentially just a manifest. 

If you're seeking a way to improve maintenance on the base source, that's probably the best way; you can literally just erase scripts and remove their reference (except for the gotcha stuff that is expected at the hard-code level, most of which is documented, thankfully).  You could even write a Python script that would make and check hashes and only re-compile A if A includes B and B has changed.

The only reason I haven't bothered doing that is because I'm lazy; I think that's how things should have been done in the first place, with a 2-3 layer file system and more organization so that you're not having to jump through 3000 lines of Other Stuff to get to that script that's referred to in your other script you're working on. 

It also would have meant that I could import TW's bug-fixes on things I haven't touched, instead of having code that's largely from 2010 except for the headers.  Like a lot of stuff on this engine (like using integers instead of unique strings to refer to item data and troops, sigh)  it would have been a better approach both for us and for them, but it's what we've got :smile:
 
There are no includes in Python. As far as I am aware!
Imports pre-interpret a file, so the imported file has to import everything necessary to be sensibly interpreted, so I need import circles and a whole bunch of useless lines in a file I created for the single one purpose to be as short as possible!

This file:
Code:
scripts.extend([
  #script_get_number_with_luck:
  # INPUT : arg0 = luck, arg1 = number
  # OUTPUT: reg0 = (number + number * luck)
  ("get_number_with_luck",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":num"),

    (val_mul, reg0, ":num"),
    (val_div, reg0, 100),
    (val_add, reg0, ":num")
  ]),

  #script_get_number_with_luck_from_troop:
  # INPUT : arg0 = troop_id, arg1 = number
  # OUTPUT: reg0 = (number + number * luck)
  ("get_number_with_luck_from_troop",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":num"),

    (troop_get_slot, reg0, reg0, slot_troop_luck),
    (val_mul, reg0, ":num"),
    (val_div, reg0, 100),
    (val_add, reg0, ":num")
  ]),

  #script_get_number_with_luck_from_party:
  # INPUT : arg0 = party_id, arg1 = number
  # OUTPUT: reg0 = (number + number * average_luck)
  ("get_number_with_luck_from_party",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":num"),

    (call_script, "script_calculate_average_party_luck", reg0),
    (val_mul, reg0, ":num"),
    (val_div, reg0, 100),
    (val_add, reg0, ":num")
  ]),

  #script_get_random_number_with_luck:
  # INPUT : arg0 = luck, arg1 = min_value, arg2 = max_value + 1
  # OUTPUT: reg0 = (random_number + random_number * luck)
  ("get_random_number_with_luck",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":min"),
    (store_script_param, ":max", 3),

    (store_random_in_range, ":rnd", ":min", ":max"),
    (val_mul, reg0, ":rnd"),
    (val_div, reg0, 100),
    (val_add, reg0, ":rnd")
  ]),

  #script_get_random_number_with_luck_from_troop:
  # INPUT : arg0 = troop_id, arg1 = min_value, arg2 = max_value + 1
  # OUTPUT: reg0 = (random_number + random_number * luck)
  ("get_random_number_with_luck_from_troop",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":min"),
    (store_script_param, ":max", 3),

    (troop_get_slot, reg0, reg0, slot_troop_luck),
    (store_random_in_range, ":rnd", ":min", ":max"),
    (val_mul, reg0, ":rnd"),
    (val_div, reg0, 100),
    (val_add, reg0, ":rnd")
  ]),

  #script_get_random_number_with_luck_from_party:
  # INPUT : arg0 = party_id, arg1 = min_value, arg2 = max_value + 1
  # OUTPUT: reg0 = (random_number + random_number * average_luck)
  ("get_random_number_with_luck_from_party",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":min"),
    (store_script_param, ":max", 3),

    (call_script, "script_calculate_average_party_luck", reg0),
    (store_random_in_range, ":rnd", ":min", ":max"),
    (val_mul, reg0, ":rnd"),
    (val_div, reg0, 100),
    (val_add, reg0, ":rnd")
  ]),

  #script_calculate_adverse_troop_lucks:
  # INPUT : arg0 = active_troop_id, arg1 = reactive_troop_id
  # OUTPUT: reg0 = relative_active_troop_luck
  ("calculate_adverse_troop_lucks",[
    (store_script_param_1, reg0),
    (store_script_param_2, ":sub"),

    (troop_get_slot, reg0, reg0, slot_troop_luck),
    (troop_get_slot, ":sub", ":sub", slot_troop_luck),
    (val_sub, reg0, ":sub")
  ]),

  #script_calculate_adverse_party_lucks:
  # INPUT : arg0 = active_party_id, arg1 = reactive_party_id
  # OUTPUT: reg0 = relative_active_party_id
  ("calculate_adverse_party_lucks",[
    (store_script_param_1, ":dom"),
    (store_script_param_2, ":sub"),

    (call_script, "script_calculate_average_party_luck", ":sub"),
    (assign, ":sub", reg0),
    (call_script, "script_calculate_average_party_luck", ":dom"),
    (val_sub, reg0, ":sub")
  ]),

  #script_calculate_average_party_luck:
  # INPUT : arg0 = party_id
  # OUTPUT: reg0 = average_luck
  ("calculate_average_party_luck",[
    (store_script_param_1, ":party_id"),

    (party_get_num_companion_stacks, ":stack_count", ":party_id"),
    (assign, reg0, 0),
    (assign, ":quotient", 0),
    (try_for_range, ":stack", 0, ":stack_count"),
      (party_stack_get_troop_id, ":stack", ":stack"),
      (troop_get_slot, ":stack", ":stack", slot_troop_luck),
      (neq, ":stack", 0), # do not count no-luckers
        (val_add, reg0, slot_troop_luck),
        (val_add, ":quotient", 1),
    (try_end),

    (try_begin),
      (neq, ":quotient", 0), # at least one lucker
        (val_div, reg0, ":quotient"),
    (try_end)
  ])
])
cannot be included via import-statement as is! The interpreter will abort on "store_script_param_1" as it is a token not defined in this file or any of its own imports.
... Easiest trick to achieve this was to append the file content to module_scripts.py. No imports needed, no lines wasted, no interpretation madness - and as it happens exactly what the #include-statement does for a C-file. True, indifferent inclusion.
If I can "just include" it, then "to include" means not "to import" and it is not a technique one trivially has to stumple across when learning the language, so I missed it, and it is neither a technique easily looked up, by the proof that I could not find it.

Damnation, did this ruffle the feathers I have not; I must be abnormally clueless to pose a problem that apparently no one else ever had!
 
You seem to be quite good at creating problems. :razz:
Finding new ways to reorganize the files doesn't really help that much. You can move around code and create hundreds of new files until the module system looks OCD-beautiful, but then you'll run into the prosaic problem of merging in new Warband patches or code OSPs. You are basically screwed, as you lost most of the reference points the merging process requires. Because no one but you has the files and code organized in the same way.
It's best to stay with the Taleworlds code organization and just get used to it, separating your code as much as you can from their code.
 
MadVader said:
It's best to stay with the Taleworlds code organization and just get used to it, separating your code as much as you can from their code.
I accept that I probably am completely oblivious to the actual meaning, but well, this sentence appears to exactly match my motivation to split the files in the first place. Modularisation, so diff/merge can be limited to as small a file as possible - module_*.py - while all included files contain 100% second or third party code.

If that were the case, however, the sense of everything before in your post were nullified. This seems implausible, so my matching likely be wrong. While in either case this is my fallacy alone, a helping hand in discerning the proper interpretation were most welcome.

... If someone could enlighten me now, how to properly include Python files in others? That would be most welcome too.
This whole situation is not only uncomfortable but also highly demotivating.
 
Back
Top Bottom