The places in the module system you need to look for are (off the top of my head):
"game_quick_start" in module_scripts.py
- This initializes a bunch of variables that are used in Quick Battle and Multiplayer
"game_custom_battle_designer" in module_presentations.py
- This handles what the Quick Battle screen looks like and starts the battle itself (together with some scripts it calls)
- Probably what you need to change the most and understand the best
"quick_battle_battle" and "quick_battle_siege" in module_mission_templates.py
- This is a list of entry points and triggers that are used during the battle (where your troops spawn, what they do, how they react to events, ...)
- There are plenty of scripts called from there, and most of the battle AI and Orders is in those scripts
"custom_battle_end" in module_game_menus.py (MUST ABSOLUTELY be #7 in the file; this is hardcoded)
- This is the debriefing screen after the battle has finished
Possibly module_troops.py and module_items.py, if you want to add custom units.
And finally, you need to build the battle scenes themselves. Run the game in Edit Mode and hit CTRL+ALT+E when you're in the scene you want to edit. It helps if you add some other menu that gets you to the scene without a battle happening on it, as this can be a bit distracting and might just screw up things in general.
Good Luck, I'd really like to play the result...