[Outdated] Xml Module System Version 0.9 (Updated 06-01-2007)

Users who are viewing this thread

codewright

Sergeant at Arms
Xml Module System Version 0.9 (Updated 06-01-2007)

The idea of this project is to improve editing of existing and creation of new content by describing it in xml rather than python code. Additionally it allows you to create small modules that only add or change content without providing the base content as well.

It has undergone rather large changes and got updated to 0.808. It generates fully functional 0.808 modules, the generated native module text files only differ slightly from the official module system. Those differences are different order of variables and dialog states, random values and some bugs in the official module system (items.txt). An important change has been made to references which will make tool development easier.

There are still some some nasty formatting/structure things that i might consider to change. But then again, xml can be transformed pretty swiftly using xslt so a structural upgrade is not too complicated. So it is of highest importance to make sure no _information_ is missing in the xml, because this can not be added by an upgrade utility. Ah, skins are missing, btw.

Requirements: Microsoft .NET 2.0 Framework

Download: XmlModuleSystem 0.9
Readme file with more detailed information

There are small examples included in the package as well. See support/examples and/or the mini-tutorial below.

There is no real need for screenshots as this is much more a data thing, but as i like screenshots myself, here are spme (old syntax!):
Pic1: The window for exporting text files from multiple xml modules.
Pic2: A syntax highlighted example of an item definition.
Pic3: Auto completition provided by Visual C# Express.
Pic4: Examples for modification statements.
Pic5: The loops.xslt in action.

Current Features:
  • Definition of game data in XML, which is more readable and very tool-friendly.
  • File-independece: different definitions (items, dialogs,...) can be mixed in one file or distributed over multiple files.
  • Simple merging of modules: Modules do not need to contain the whole game data but can only define the data that is new.
  • conversation tool to create a XML module from python code
  • Complete xml schema allows validation and auto-completition provided by xml editors and parsers.
  • automatic "slot" handling: just declare and use them, numbers will be associated automatically
  • automatic "category" handling: just mark a party, troop or quest with spawnpoints, towns, merchants,... and the exporter will put them into the right order.
  • more flexible modification operations like redefine, partial overwrite and removal of items while merging modules.
  • xslt processor tool for user defined xml transformations (currently there is the loops.xslt bundled)

Planned Features:
  • resources included: module brings all the resources that it adds to the game and the export function makes sure all _needed_ resources are in the m&b module.
  • support for skins and music
  • an alternative to the current code syntax which can be used in code blocks


Mini-Tutorial: How to add an item
Step one: Create a new module file.
Simply create a new file in the Modules directory called "my-module.xml" and open it in a text or xml editor. Then put in the following text:
Code:
<?xml version="1.0"?>
<Module Version="0.4" xmlns="urn:mount-and-blade:xml-module-system:0.4">
	<Name>My Module</Name>
	<Version>1.0</Version>
	<Definitions>
		<!-- your definitions go here -->
	</Definitions>
</Module>
Step two: Add your definition.
Now that you have your own single-file module you can add any definition like items, dialogs, troops and so on. As we want to add an item in this tutorial you will want to have a look at the items.xml in the native module. There you can copy/paste the basic structure of any item to your new module and modify it to your needs. The new definition needs to be put inside the Definitions tag. For example, a helmet may look like this:
Code:
	<Item Id="my_helmet">
		<Name>My New Helmet</Name>
		<Type>head_armor</Type>
		<Value>433</Value>
		<MeshAssignments>
			<MeshAssignment>
				<Mesh Resource="lamellar_helmet_a"/>
			</MeshAssignment>
		</MeshAssignments>
		<ItemModifiers>
			<ItemModifierList RefId="cloth"/>
		</ItemModifiers>
		<Stats>
			<Weight>2</Weight>
			<Abundance>100</Abundance>
			<HeadArmor>19</HeadArmor>
		</Stats>
	</Item>

Step three: Export to text module.
Now that your new xml module is done you need to generate the text files used by Mount&Blade. To do that you need to open up the XmlModuleSystem.exe. In the list on the left side you should see the official module and your new module, called "My Module". Simply enable both by checking the boxes and click export. The text files will be put into the "export" folder. To use them in M&B you need to copy them into a module under the Modules folder of Mount&Blade.

Attention: By default savegames are not compatible with exports of the xml module system, because the order of dialog_states and variables is different. If you want to get best compatibility possible you may want to copy the predef files from support/predef into the same directory as the XmlModuleSystem.exe resides and reexport the module. The files provided are the same files as the official module system (0.80:cool: exported and will be used to use the same order of definitions. This MAY cause problems with many changes to the native module, but should work fine most of the time.



Version 0.9
(6 Jan 2006)

* updated to MB Version 0.808
* changed naming schema to Pascallike (i.e. AnIdentifier)
* explicit difference between reference and definition for most entities
  (no longer nested definitions)

complete ChangeLog

i appreciate any comments, hints, bug reports, suggestions and proposals about the xml structure, the whole idea and everything else.
Have fun :wink:
 
Wow. I'm not modding any more but this looks like it will be a huge help.  :smile:
 
thanks dagorkan :wink:

i forgot to mention a few things:
first, reordering of items to comply with the begin/end "groups" is not done yet. say, if you add a town in a module it will not be automatically inserted into the "town_begin"/"town_end" block/group. there is just some sorting algorithm missing that does preserve a minimum order so at least the hardcoded objects stay at the beginning. the alternative would be to always make sure hardcoded items are at the beginning by hardcoding this rule :razz: (hardcoding is so contagious)

second, there are some differences in the exported text files compared to the original module. reasons for this are:
  • different order of variables, dialog_states and (not yet) definitions.
  • some bugs in the official module system. i identified the following:
    • skills are connected by bit-or which can lead to higher values (knows_common|knows_riding_2 -> riding 3)
    • hitpoints are set without specification (in fact, some items have very random hitpoints as far as i can see; seems to be a not-masked bit operation)
    • weapon length bit 0 is set to 0 somewhere, so all items have even weapon lengths even if not specified.
  • weapon proficiencies (see below)

third, (oops, i wanted to change this before release :oops:) weapon proficiencies are not in except the base wp (python wp(50) for example). and this base wp is not randomized like in the official module system. if i may add a side note here: i don't think the module-export is the best place for randomizing values.

anyway, thats all i can think of now. keep in mind that it's an alpha release so some bugs are still "allowed" :wink:
 
So, it actually covers every other module file besides the skins file? Wow, quite a first release of it. I'm sure nobody expects it to be perfect at the first release. :mrgreen:

Will definitely be checking this out tomorrow when I get a chance; I will be quite busy today.
 
That's some good work there :smile: MY two cents: I would like it a lot more with a GUI front end. That seems like a lot to type in to define a single item, unless all the tags are there by default. I haven't used yet to see for sure.
 
Fujiwara said:
That's some good work there :smile: MY two cents: I would like it a lot more with a GUI front end. That seems like a lot to type in to define a single item, unless all the tags are there by default. I haven't used yet to see for sure.
Well, that's one major benefit in my mind: just about every programming language which is even close to up-to-date has an XML parser available. So, setting up a new Unofficial Editor type program (friendly GUI and all that) for editing the XML files would be much easier than setting one up to parse the python files.
 
Yep, building editors or other tools for the xml stuff is way easier than for python code or even text files. Additionally, because i switched to .net already there is a complete object model for the definitions available already.

Even without an explicit m&b editor you can use the comfort some xml editors provide when they know the scheme of the file. I will look into this and try to find an easy and free editor. Most IDEs like the visual c# express and eclipse provide such editors, but it seems like i need to tweak the schema definitions a little until this works.
 
As editied above, the new version 0.2 fixes the missing weapon proficiencies (err, am I blind or does the official module really only use wp(...)?) and the missing reordering of quests, parties and troops according to their groups. So you can now add an archer with superb archery skills, put his category to "regular" and instead of appearing at the end he will be inserted at the end of the regular group (say after enemy_hero if i'm not mistaken). This makes sure scripts iterating over such groups dont break.
 
Is it possible to mix the xml with existing Python code? I'm interested in trying this out, but the shear amount of time involved in porting over is daunting. I need to work on actually building my mod, so time spent doing other things isn't well used.
 
There are weapon type specific versions of wp(x) also. Do you see those?
yes, but are they actually used anywhere? i only found wp() calls, even the archers only have wp() although they should have a way higher archery value.

Is it possible to mix the xml with existing Python code? I'm interested in trying this out, but the shear amount of time involved in porting over is daunting. I need to work on actually building my mod, so time spent doing other things isn't well used.
Please try the python2xml scripts. Just copy them into your module system directory and execute the xml_convert.bat. If you havent changed tooo much it should work, in case there are problems/error please report them to me. The result is located in xml/native which of course is the wrong name if you changed things. Additionally you will need the items-header.xml and troops-header.xml to be complete. They are contained in the provided native module so you will not need to copy them over if you strip your module and export it together with the native module.
 
codewright said:
yes, but are they actually used anywhere? i only found wp() calls, even the archers only have wp() although they should have a way higher archery value.

I don't know about vanilla, but I use them quite a bit for Onin no Ran.
 
Just uploaded a new version. Now the Xml Schema is complete and you can use the auto-completition function of xml editors like the one in Visual C# Express. I added a screenshot in case someone does not exactly know what this means in the end :wink:. You may need to be online to use this functions although the whole schema is within the zip file as well so you might be able to tell the editor to use the local files instead. (try the properties view in VC#E)

Be aware that the schema version number is used in the xml files and the exporter tool. That means if the xml file is not the same version the exporter may not work. Sooner or later there will be a conversation ("upgrade") tool to convert old xml to newer but this does not exist yet. As this is still alpha i will still be a bit more aggressive concerning changes. If the schema is good enough and there is some 1.0 release the schema will (hopefully) only be changed with new M&B releases. So although i highly encourage you to try the xml module, be prepared to react to some changes still.

On the other hand, feel free concerning proposals about changes. As long as this is not a 1.0 i will consider all changes that make sense.

Just to name a few things i'm uncertain about:
  • operations: one could put them into one tag <operation code="bla" param1="blub" param2="bleh"/>, but thats not so much xml style
  • flags are handled differently in some places, i.e. sometimes there is a "flags" tag because there are many flags possible and sometimes they are in the definition itself, because there are only one or two flags.
  • tag name vs. "name" attribute: flags are named by their tag ("<melee/>") but skills are named in an attribute ("<skill name="ironfist"/>")
  • ...

I know most of the things are about style and not functionality, but an intuitive and uniform style can speed up things pretty often :wink:
 
Fujiwara said:
codewright said:
yes, but are they actually used anywhere? i only found wp() calls, even the archers only have wp() although they should have a way higher archery value.

I don't know about vanilla, but I use them quite a bit for Onin no Ran.

anyway, it's in now, so you won't miss it there. i just dont know whether the python2xml converter will do this completely correct because i didn't test it directly. but as far as i can see it should work :wink:
 
Ok, I found so bugs I'm sure you want to know about:

http://i46.photobucket.com/albums/f127/texan_otaku/xml_editor_bug01.jpg
http://i46.photobucket.com/albums/f127/texan_otaku/xml_editor_bug02.jpg
http://i46.photobucket.com/albums/f127/texan_otaku/xml_editor_bug03.jpg

For some (maybe all, I haven't looked at them all yet) it doesn't seem to finish. In xml_editor_bug03, at the bottom, it stopped maybe 1/3 of the way through the python file. Here's the error message:

Code:
A subdirectory or file xml already exists.
A subdirectory or file xml\Onin_no_Ran already exists.
Traceback (most recent call last):
  File "xml_convert_dialogs.py", line 42, in ?
    convertDialog( dialog )
  File "xml_convert_dialogs.py", line 23, in convertDialog
    convertScript( fOut, dialog[2], "condition" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 228, in convertScript
    convertPlainScript( fOut, script, indent+"\t" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 193, in convertPlainScript
    v = convertCategoryAccess( v )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 179, in convertCategoryAccess
    raise Exception( "Unknown category access: "+val )
Exception: Unknown category access: weaponsmiths_begin
Traceback (most recent call last):
  File "xml_convert_factions.py", line 39, in ?
    convertFaction( faction )
  File "xml_convert_factions.py", line 16, in convertFaction
    fOut.write( "\t\t\t<relation value=\"%s\">\n" % (relation[1].strip()) )
IndexError: string index out of range
Traceback (most recent call last):
  File "xml_convert_game_menus.py", line 40, in ?
    convertFaction( gameMenu )
  File "xml_convert_game_menus.py", line 9, in convertFaction
    convertScript( fOut, gameMenu[4], "script" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 228, in convertScript
    convertPlainScript( fOut, script, indent+"\t" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 191, in convertPlainScript
    v = convertIdToReference( op[i] )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 391, in convertIdToReference
    if( id.startswith( "reg(" ) ):
AttributeError: 'list' object has no attribute 'startswith'
Traceback (most recent call last):
  File "xml_convert_items.py", line 122, in ?
    convertItem( item )
  File "xml_convert_items.py", line 45, in convertItem
    fOut.write( "\t\t<type>%s</type>\n" % type )
UnboundLocalError: local variable 'type' referenced before assignment
Traceback (most recent call last):
  File "xml_convert_scenes.py", line 71, in ?
    convertScene( scene )
  File "xml_convert_scenes.py", line 23, in convertScene
    if( scene[2] != "none" ):
IndexError: list index out of range
Traceback (most recent call last):
  File "xml_convert_scripts.py", line 27, in ?
    convertScript( script )
  File "xml_convert_scripts.py", line 7, in convertScript
    convertPlainScript( fOut, script[1], "\t\t" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 193, in convertPlainScript
    v = convertCategoryAccess( v )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 179, in convertCategoryAccess
    raise Exception( "Unknown category access: "+val )
Exception: Unknown category access: heroes_begin
Traceback (most recent call last):
  File "xml_convert_simple_triggers.py", line 23, in ?
    convertSimpleTrigger( simpleTrigger )
  File "xml_convert_simple_triggers.py", line 6, in convertSimpleTrigger
    convertTrigger( fOut, simpleTrigger, "\t", 1 )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 312, in convertTrigger
    convertScript( fOut, trigger[1], "script", indent+"\t" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 228, in convertScript
    convertPlainScript( fOut, script, indent+"\t" )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 193, in convertPlainScript
    v = convertCategoryAccess( v )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 179, in convertCategoryAccess
    raise Exception( "Unknown category access: "+val )
Exception: Unknown category access: cities_end
Traceback (most recent call last):
  File "xml_convert_triggers.py", line 24, in ?
    convertTriggerI( trigger )
  File "xml_convert_triggers.py", line 6, in convertTriggerI
    convertTrigger( fOut, trigger, "\t", 1 )
  File "C:\Documents and Settings\Matt\My Documents\M&B\Tools\XmlModuleSystem-0.
3\python2xml\xml_convert.py", line 318, in convertTrigger
    if( trigger[0].startswith( "ti_val(" ) ):
AttributeError: 'list' object has no attribute 'startswith'
Traceback (most recent call last):
  File "xml_convert_troops.py", line 143, in ?
    convertTroop( troop )
  File "xml_convert_troops.py", line 110, in convertTroop
    upgrades = findUpgrades( troop[0].strip() );
  File "xml_convert_troops.py", line 12, in findUpgrades
    if( p[1] == troopId ):
IndexError: list index out of range
Press any key to continue . . .
 
ah thanks :wink:
looks like i got sth to do here. that is what i meant by saying parsing python code is "not very reliable".

the xml errors seem to arise from the unfinished xml files so they should not be the cause of the trouble.
just give me a little time ...
 
No problem, glad to be of help. This is some good work. I can see it really speeding along mod making once finished.  :grin:
 
As very first note: I am _parsing_ the python code. That means i do not evaluate it so user made constants or other programmatical constructs are not recognized at all (in most cases). I did some special handling of constants in the native module (i.e. pilgrim_disguise) but thats only concerning the native module. So in case you use something like that you will need to replace them with their values. (or tell me, in case you use it very often and do not want to loose the modularity).


Ok, lets go through the exceptions.
The first one "Unknown category access: weaponsmiths_begin" is thrown because you have additional begin/end-blocks or as i call them "categories". You will need to add them to the python2xml.

-- how to add a category (begin/end pair) --

The idea of begin/end pairs is to have two values which you can use to iterate over a certain part of quests, troops or parties. In the xml module system this is done by "categories". Each quest, troop or party can be in a category and categories can be accessed in scripts by "category-begin:troop:merchants/weapon" for example.

There are two things you need to add in order to convert them automatically. First you need to add a mapping constant name (that is accessed in scripts) to category name. This is done by adding the constant-pairs name to the "convertCategoryAccess" function in xml_convert.py line 143. for example if you want to add "weaponsmiths_begin/end" as a troop category simply add

Code:
elif( cat == "weaponsmiths" ):
    cat = "troop:same-or-new-category-name"

to the if/elif block for troops (or anywhere else). You may want to add some hierarchy but that is not necessary (the / is more than just a symbol there). now each time a script referres to this constant it is replaced by "category-begin/end:troop:same-or-new-category-name".

Now we will need to identify the quests, troops and parties that should go into certain categories. this is done in the respective xml_convert_*.py file. In this case we need to edit the xml_convert_troops.py and add a rule to the block beginning at line 25. "some rule" means you can use all means to find out to which category this troop should belong to. I check the id which is the first (0) field in the array for some substrings or a complete name. The category is not reset automatically so you can check for the first id that should be in the category, set the category and then check for the last id and reset the category (this is done for category "regular" for example.) The category you map to must be the same as you used above except the type identifier (troop:smile:.

Dang, the new category is ready to go :wink:

-------------------------

The second exception concerning factions is because a relation does not have a second value. Normally a factions last value is an array having 2-tupels in there like "[("player_faction",0.0)]". For some reason the second value does not exist. This might be caused by some constants you use as said above or some parsing error. Do you use some special constructs in the factions module that are different from the vanilla one?


The third exception is about a script (4th field) in a game menu. There must be some parameter that is recognized as a list. As tupels are lists for me that may be a formula like (4+2) as this is a one-element list for me. If you use formulas there often, please tell me, then i'll add some support there, if not try removing the parentheses at least. Looks like i'll need to change there something anywhay as the formula will end up in the xml but will not be handled there (as far as i know). So if you can remove the parentheses but do not want to resolve the formula then i'll need some basic calculation support in the exporter.


The fourth is easy: there is an item without a itp_type_* flag, but i need this flag. So in case it doesn't matter take itp_type_goods or whatever.

Next one is about scenes, looks like a parse problem because it does not find a third entry in a scenes entry. Please check whether there are some array/tupel constructs in the scenes array that do not belong there.

Next two are categories again (heroes, cities), see above.

Now there is a trigger which seems to have a formula in the first field. Looks like i only have formula handling for the third field, so try removing the parentheses if that is possible. Btw, even the third field doesn't really "handle" formulas, it just takes the content of the first parentheses if there are any.
So please tell me how much formula handling you _need_ or whether you can live with resolving them.

The last exception is about upgrades. seems like there is an upgrade* call with only one parameter. This confuses the converter :wink:


As you can see, it is a fragile thing to parse code and you seem to have made major changes. I hope this will help you to solve some of the problems yourself otherwise just tell me :wink:.
 
If you like, I can supply you with my source code, since I rely a great deal on scripts and formulas and variables. It would seem that a great deal more is going on in my code than in the vanilla. :smile:
 
Back
Top Bottom