OSP Code Optimisation Module system hackery: reporting of variable issues (MSH:301)

Users who are viewing this thread

In this installment of "kt0 hacks the module system in devious ways" we explore the topic of tracking down variable warnings and errors.  The typical disclaimers apply as always:
- I've tested things locally but that doesn't mean there aren't any bugs.
- When applying these changes, you have to be very precise.
- If you're squeamish about real programming, look away and don't try to apply any of these changes.
- I claim no merchantability for these hacks. 

Motivation
I hack stuff up--a lot, and I occasionally miss variable warnings/errors, especially when I'm half drunk (i.e., most of the time).  When the game generates variable warnings and errors (unassiged local, unassigned global, unused local), its default granularity is per file which is convenient if your module system files are significantly less than 1,000 lines long (mine aren't).  This set of hacks is intended to give useful information on where to look for such issues and is especially useful when the variable in question is something appearing regularly like ":faction" or ":center_no". 

Procedure
BACKUP YOUR MODULE SYSTEM CODE BEFORE MAKING ANY CHANGES!

We proceed file by file in the process_*.py methods.  The basic idea is to pass a meaningful location string for each place that we might report a warning/error for a local variable which would be straightforward except that it touches a lot of the process function definitions. 

process_operations.py contains the root piece of the changes.  get_variable is the function responsible for displaying the error concerning usage of an unassigned variable both global and local.  we add a calling script string parameter to the function so we can report it later:
def get_variable(variable_string,variables_list,variable_uses, calling_script):
  found = 0
  result = -1
  var_string = variable_string[1:]
  for i_t in xrange(len(variables_list)):
    if var_string == variables_list[i_t]:
      found = 1
      result = i_t
      variable_uses[result] = variable_uses[result] + 1
      break
  if not found:
    if (variable_string[0] == '$'):
      variables_list.append(variable_string)
      variable_uses.append(0)
      result = len(variables_list) - 1
      print "WARNING: Usage of unassigned global variable: " + variable_string + " in script '" + calling_script + "'"
    else:
      print "ERROR: Usage of unassigned local variable: " + variable_string + " in script '" + calling_script + "'"
  return result

Following this is a lot of cleanup to pass the appropriate variable down to the function.  In the same file, make the following changes (assorted by function):
def process_param(param,global_vars_list,global_var_uses, local_vars_list, local_var_uses, tag_uses, quick_strings, calling_script):
  result = 0
  if (type(param) == types.StringType):
    if (param[0] == '$'):
      check_varible_not_defined(param[1:], local_vars_list)
      result = get_variable(param, global_vars_list,global_var_uses, calling_script)
      result |= opmask_variable
    elif (param[0] == ':'):
      check_varible_not_defined(param[1:], global_vars_list)
      result = get_variable(param, local_vars_list,local_var_uses, calling_script)
      result |= opmask_local_variable
    elif (param[0] == '@'):
      result = insert_quick_string_with_auto_id(param[1:], quick_strings)
      result |= opmask_quick_string
    else:
      result = get_identifier_value(param.lower(), tag_uses)
      if (result < 0):
        print "ERROR: Illegal Identifier:" + param
  else:
    result = param
  return result


def save_statement(ofile,opcode,no_variables,statement,variable_list,variable_uses,local_vars_list,local_var_uses,tag_uses,quick_strings, calling_script):
  if no_variables == 0:
    lenstatement = len(statement) - 1
    if (is_lhs_operation(opcode) == 1):
      if (lenstatement > 0):
        param = statement[1]
        if (type(param) == types.StringType):
          if (param[0] == ':'):
            add_variable(param[1:], local_vars_list, local_var_uses)
  else:
    lenstatement = 0
  ofile.write("%d %d "%(opcode, lenstatement))
  for i in xrange(lenstatement):
    operand = process_param(statement[i + 1],variable_list,variable_uses,local_vars_list,local_var_uses,tag_uses,quick_strings, calling_script)
    ofile.write("%d "%operand)


def save_statement_block(ofile,statement_name,can_fail_statement,statement_block,variable_list, variable_uses,tag_uses,quick_strings, calling_script):
  local_vars = []
  ... # more stuff here
    save_statement(ofile,opcode,no_variables,statement,variable_list,variable_uses,local_vars, local_var_uses,tag_uses,quick_strings, calling_script)
  if (store_script_param_1_uses > 1):
    print "WARNING: store_script_param_1 is used more than once:" + statement_name
  if (store_script_param_2_uses > 1):
    print "WARNING: store_script_param_2 is used more than once:" + statement_name
  i = 0
  while (i < len(local_vars)):
    if (local_var_uses[ i] == 0 and not(local_vars[ i].startswith("unused"))):
      print "WARNING: Local variable never used: " + local_vars[ i] + " in script '" + calling_script + "'"
    i = i + 1

# kt0:  this is one of the top level save methods that's used in a couple places (probably why it appears here
# rather than elsewhere).  this is an example of building the name to pass into the modified functions; in this
# case, just the ID of the trigger since they don't have names.
def save_simple_triggers(ofile,triggers,variable_list, variable_uses,tag_uses,quick_strings):
  ofile.write("%d\n"%len(triggers))
  trigger_id = 0
  for trigger in triggers:
    ofile.write("%f "%(trigger[0]))
    save_statement_block(ofile,0,1,trigger[1]  , variable_list, variable_uses,tag_uses,quick_strings, "trigger " + str(trigger_id) )
    ofile.write("\n")
    trigger_id += 1
  ofile.write("\n")

The rest of it (broken up per file) is passing meaningful names in as the added parameter.

process_dialogs.py:
def save_triggers(variable_list,variable_uses,triggers,tag_uses,quick_strings):
  file = open(export_dir + "triggers.txt","w")
  file.write("triggersfile version 1\n")
  file.write("%d\n"%len(triggers))
  for i in xrange(len(triggers)):
    trigger = triggers[ i]
    file.write("%f %f %f "%(trigger[trigger_check_pos],trigger[trigger_delay_pos],trigger[trigger_rearm_pos]))
    trigger_id = "trigger " + str(i)
    save_statement_block(file,0,1,trigger[trigger_conditions_pos]  , variable_list, variable_uses,tag_uses,quick_strings, trigger_id)
    save_statement_block(file,0,1,trigger[trigger_consequences_pos], variable_list, variable_uses,tag_uses,quick_strings, trigger_id)

...


def save_sentences(variable_list,variable_uses,sentences,tag_uses,quick_strings,input_states,output_states):
  file = open(export_dir + "conversation.txt","w")
  file.write("dialogsfile version 1\n")
  file.write("%d\n"%len(sentences))
  # Create an empty dictionary
  auto_ids = {}
  for i in xrange(len(sentences)):
    sentence = sentences[ i]
    try:
      dialog_id = create_auto_id2(sentence,auto_ids)
      file.write("%s %d %d "%(dialog_id,sentence[speaker_pos],input_states[ i]))
      save_statement_block(file, 0, 1, sentence[sentence_conditions_pos], variable_list,variable_uses,tag_uses,quick_strings, dialog_id+" condition block")

      file.write("%s "%(string.replace(sentence[text_pos]," ","_")))
      if (len(sentence[text_pos]) == 0):
        file.write("NO_TEXT ")
      file.write(" %d "%(output_states[ i]))
      save_statement_block(file, 0, 1, sentence[sentence_consequences_pos], variable_list,variable_uses,tag_uses,quick_strings, dialog_id+" consequence block")
      file.write("\n")
    except:
      print "Error in dialog line:"
      print sentence
  file.close()

process_game_menus.py:
def save_game_menu_item(ofile,variable_list,variable_uses,menu_item,tag_uses,quick_strings):
  ofile.write(" mno_%s "%(menu_item[0]))
  save_statement_block(ofile,0, 1, menu_item[1], variable_list, variable_uses,tag_uses,quick_strings, "mno_"+menu_item[0]+" condition block" )
  ofile.write(" %s "%(string.replace(menu_item[2]," ","_")))
  save_statement_block(ofile,0, 1, menu_item[3], variable_list, variable_uses,tag_uses,quick_strings, "mno_"+menu_item[0]+" consequence block" )
  door_name = "."
  if (len(menu_item) > 4):
    door_name = menu_item[4]
  ofile.write(" %s "%(string.replace(door_name," ","_")))


def save_game_menus(variable_list,variable_uses,tag_uses,quick_strings):
  ofile = open(export_dir + "menus.txt","w")
  ofile.write("menusfile version 1\n")
  ofile.write(" %d\n"%(len(game_menus)))
  for game_menu in game_menus:
    ofile.write("menu_%s %d %s %s"%(game_menu[0],game_menu[1],string.replace(game_menu[2]," ","_"),game_menu[3]))
    save_statement_block(ofile,0,1, game_menu[4]  , variable_list, variable_uses,tag_uses,quick_strings, "menu_"+game_menu[0] )
    menu_items = game_menu[5]
    ofile.write("%d\n"%(len(menu_items)))
    for menu_item in menu_items:
      save_game_menu_item(ofile,variable_list,variable_uses,menu_item,tag_uses,quick_strings)
    ofile.write("\n")
  ofile.close()

process_mission_tmps.py:
def save_triggers(file,template_name,triggers,variable_list,variable_uses,tag_uses,quick_strings):
  file.write("%d\n"%len(triggers))
  for i in xrange(len(triggers)):
    trigger = triggers[ i]
    file.write("%f %f %f "%(trigger[trigger_check_pos],trigger[trigger_delay_pos],trigger[trigger_rearm_pos]))
    save_statement_block(file, 0, 1, trigger[trigger_conditions_pos]  , variable_list,variable_uses,tag_uses,quick_strings, "mission template " + str(i) + " condition block" )
    save_statement_block(file, 0, 1, trigger[trigger_consequences_pos], variable_list,variable_uses,tag_uses,quick_strings, "mission template " + str(i) + " consequence block" )
    file.write("\n")
  file.write("\n")

process_scripts.py:
def save_scripts(variable_list,variable_uses,scripts,tag_uses,quick_strings):
  file = open(export_dir + "scripts.txt","w")
  file.write("scriptsfile version 1\n")
  file.write("%d\n"%len(scripts))
  temp_list = []
  list_type = type(temp_list)
  for i_script in xrange(len(scripts)):
    func = scripts[i_script]
    if (type(func[1]) == list_type):
      file.write("%s -1\n"%(convert_to_identifier(func[0])))
      save_statement_block(file,convert_to_identifier(func[0]), 0,func[1], variable_list,variable_uses,tag_uses,quick_strings, convert_to_identifier(func[0]) )
    else:
      file.write("%s %f\n"%(convert_to_identifier(func[0]), func[1]))
      save_statement_block(file,convert_to_identifier(func[0]), 0,func[2], variable_list,variable_uses,tag_uses,quick_strings, convert_to_identifier(func[0]) )
    file.write("\n")
  file.close()

process_simple_triggers.py:
def save_simple_triggers(variable_list,variable_uses,triggers,tag_uses,quick_strings):
  file = open(export_dir + "simple_triggers.txt","w")
  file.write("simple_triggers_file version 1\n")
  file.write("%d\n"%len(simple_triggers))
  for i in xrange(len(simple_triggers)):
    simple_trigger = simple_triggers[ i]
    file.write("%f "%(simple_trigger[0]))
    save_statement_block(file,0, 1, simple_trigger[1]  , variable_list,variable_uses,tag_uses,quick_strings, "simple trigger "+str(i) )
    file.write("\n")
  file.close()

process_tableau_materials.py:
def save_tableau_materials(variable_list,variable_uses,tag_uses,quick_strings):
  ofile = open(export_dir + "tableau_materials.txt","w")
  ofile.write("%d\n"%(len(tableaus)))
  tableau_id = 0
  for tableau in tableaus:
    ofile.write("tab_%s %d %s %d %d %d %d %d %d"%(tableau[0], tableau[1], tableau[2], tableau[3], tableau[4], tableau[5], tableau[6], tableau[7], tableau[8]))
    save_statement_block(ofile, 0, 1, tableau[9], variable_list, variable_uses, tag_uses, quick_strings, "tableau "+str(tableau_id) )
    ofile.write("\n")
    tableau_id += 1
  ofile.close()

Hopefully I haven't missed anything. 

Results
Now you can merilly add local and global variable errors and the compilation process will give you proper readouts such as this:
Initializing...
Compiling all global variables...
Module.ini updated.
Exporting strings...
Exporting skills...
Exporting tracks...
Exporting animations...
Exporting meshes...
Exporting sounds...
Exporting skins...
Exporting map icons...
Creating new tag_uses.txt file...
Creating new quick_strings.txt file...
Exporting faction data...
Exporting item data...
Exporting scene data...
Exporting troops data
Exporting particle data...
Exporting scene props...
Exporting tableau materials data...
ERROR: Usage of unassigned local variable: :blarg in script 'tableau 3'
Exporting presentations...
Exporting party_template data...
Exporting parties
Exporting quest data...
Exporting scripts...
ERROR: Usage of unassigned local variable: :center_no in script 'spawn_bandits'
Exporting mission_template data...
ERROR: Usage of unassigned local variable: :blarg_cond in script 'mission template 7 condition block'
ERROR: Usage of unassigned local variable: :blarg_cons in script 'mission template 7 consequence block'
ERROR: Usage of unassigned local variable: :blarg_cond in script 'mission template 9 condition block'
ERROR: Usage of unassigned local variable: :blarg_cons in script 'mission template 9 consequence block'
ERROR: Usage of unassigned local variable: :blarg_cond in script 'mission template 7 condition block'
ERROR: Usage of unassigned local variable: :blarg_cons in script 'mission template 7 consequence block'
Exporting game menus data...
ERROR: Usage of unassigned local variable: :blarg_cond in script 'mno_start_female condition block'
ERROR: Usage of unassigned local variable: :blarg_cons in script 'mno_start_female consequence block'
exporting simple triggers...
ERROR: Usage of unassigned local variable: :blarg in script 'simple trigger 14'
WARNING: Local variable never used: blarg2 in script 'simple trigger 14'
exporting triggers...
exporting dialogs...
ERROR: Usage of unassigned local variable: :blarg_cond in script 'dlga_prisoner_chat_6:close_window condition block'
ERROR: Usage of unassigned local variable: :blarg_cons in script 'dlga_prisoner_chat_6:close_window consequence block'
Checking global variable usages...

______________________________

Script processing has ended.
Press any key to exit. . .

Future Work
MSH:302 titled "Module system hackery:  additional reporting of local variable issues" will explore the possibility of adding statement lines in addition to caller lines.


ps)  Oh yeah, almost forgot:  BLAM.
 
That is some incredible stuff Kto I've always freaked out when I get a process.py type error because it never points you to the actual error.  You are doing some awesome work with this stuff.
 
Back
Top Bottom