DtheHun
Sergeant Knight
Object
Mimic the phonation of the sounds and facial expressions of a recorded speech.
Demonstration
Skill requirements (topics not covered in this tutorial)
- loading brf file
- add new race
- spawn troop
- set up dialog or trigger event in scene
Talkress
A "race" with only 8 face keys based on a slightly changed vanilla head mesh, completed with teeth and tongue. The usual face trait frames can be added later.
You can download and use this head model. talkress.brf Set "talkress_head" mesh to be used by the race of the speaking troop.
Under the hood
The first eight shape keys (variations of the base head model) are scalable separately by code. Refreshing the head display in scenes can be achieved by helm equip&unequip, so it's done once per frame. The eight scalable shape IDs and their values are both in the [0-7] range, so I load them frame by frame into two arrays containing nuber lines where the values are separated by their decimal places.
M, B, P sounds have no keys, they are formed with closed mouth
Code samples
Loading speech sounds
These two lines (module_dialogues.py) add "F" and "R" sound to the playlist.
(call_script, "script_set_keys_and_values", 2, 4, 7), #F
(call_script, "script_set_keys_and_values", 2, 24, 50), #R
2: DELAY - 2 frame steps from the previous sound +(2*40ms or 2*25[1/s])
2: ID#2 (F,V)
4: ID#1 (E,R,K,G,H)
5: value#2 - 5 in [0-7] range: R sound is less "open" than E, a little gap between the teeth is enough (use a mirror)
0: value#1 - 0 in [0-7] range: switches off the previously used shape for F
The second line contains a pair of IDs and values. It zeroes out the previous "F" sound (ID=4, value=0) to shape the mouth to an "R" sound (ID=2, value=5) in 2 frame later.
I use 2 frame steps frequently. That's usually a good tempo, and my code calculates average shapes for the in-between frames to smooth out the animation. When the tempo must be changed (other than 2 frame steps needed) to be in synchron with the audio, you have to make the changes gradually to keep the animation smooth.
IDs and values are construed from right to left (<-), if there are less values than IDs, then the value of the undefined ID will be 0.
IMPORTANT to do NOT start these number lines with 0: You can change the order of IDs (don't forget to change the values too), and skip the starting zeros of values.
For example, the first call - which resets the default morph values -
specifies all the 8 IDs, but gives them only 7 values.
Therefore the value of the leftmost id (7) will be set to 0. (Blink=0 -> start dialogue with eyes opened)
You can find more working code samples in WarDrobe ModuleSystem.
Corrections and suggestions are welcome.
Edit:
module_skins.py code sample added
list of sounds order corrected
note inserted about M,B,P sounds
Mimic the phonation of the sounds and facial expressions of a recorded speech.
Demonstration
Skill requirements (topics not covered in this tutorial)
- loading brf file
- add new race
- spawn troop
- set up dialog or trigger event in scene
Talkress
A "race" with only 8 face keys based on a slightly changed vanilla head mesh, completed with teeth and tongue. The usual face trait frames can be added later.
You can download and use this head model. talkress.brf Set "talkress_head" mesh to be used by the race of the speaking troop.
Under the hood
The first eight shape keys (variations of the base head model) are scalable separately by code. Refreshing the head display in scenes can be achieved by helm equip&unequip, so it's done once per frame. The eight scalable shape IDs and their values are both in the [0-7] range, so I load them frame by frame into two arrays containing nuber lines where the values are separated by their decimal places.
M, B, P sounds have no keys, they are formed with closed mouth
Code samples
Code:
[anyone,"talkress_talk2", [],
"...From Me? Of course from Me! Ha-ha-hah!", "talkress_talk3", [
#Lip-Sync
(assign, "$lipsync_agent", "$g_talk_agent"),
#clear temp troop arrays - use A for ID, B for value
(assign, "$talk_length", 250),
(store_add, ":last_frame_plus_one", "$talk_length", 1),
(try_for_range, ":slot_no", 0, ":last_frame_plus_one"),
(troop_set_slot, "trp_temp_array_a", ":slot_no", -1),
(troop_set_slot, "trp_temp_array_b", ":slot_no", -1),
(try_end),
(assign, "$talk_length", 0),
(call_script, "script_set_keys_and_values", 1, 76543210, 3300000), #XX
(call_script, "script_set_keys_and_values", 2, 5, 5), #H Sad/Happy5 (closed mouth move ~ WUT???)
(call_script, "script_set_keys_and_values", 2, 5, 6), #M Sad/Happy6
(call_script, "script_set_keys_and_values", 2, 5, 7), #M Sad/Happy7
(call_script, "script_set_keys_and_values", 2, 5, 3), #M??? Sad/Happy3
(call_script, "script_set_keys_and_values", 2, 4, 7), #F
(call_script, "script_set_keys_and_values", 2, 24, 50), #R
(call_script, "script_set_keys_and_values", 2, 32, 70), #O
(call_script, "script_set_keys_and_values", 2, 3, 0), #M
(call_script, "script_set_keys_and_values", 2, 63, 70), #M Angry/Wonder7
(call_script, "script_set_keys_and_values", 2, 2, 7), #I'?
(call_script, "script_set_keys_and_values", 2, 7, 7), #- Blink7
(call_script, "script_set_keys_and_values", 2, 7, 5), #- Blink5
(call_script, "script_set_keys_and_values", 2, 72, 0), #- Blink0
(call_script, "script_set_keys_and_values", 18, 63, 45), #O Angry/Wonder4
(call_script, "script_set_keys_and_values", 2, 543, 770), #F Sad/Happy7
(call_script, "script_set_keys_and_values", 2, 13, 53), #K
(call_script, "script_set_keys_and_values", 2, 31, 50), #O
(call_script, "script_set_keys_and_values", 2, 23, 70), #RZ
(call_script, "script_set_keys_and_values", 4, 42, 70), #F
(call_script, "script_set_keys_and_values", 2, 24, 50), #R
(call_script, "script_set_keys_and_values", 2, 32, 70), #O
(call_script, "script_set_keys_and_values", 2, 3, 0), #M
(call_script, "script_set_keys_and_values", 2, 3, 0), #M
(call_script, "script_set_keys_and_values", 2, 2, 7), #I'
(call_script, "script_set_keys_and_values", 6, 475, 774), #V Blink7 Sad/Happy4 (start to smile)
(call_script, "script_set_keys_and_values", 2, 4075, 725), #A' Blink5 Sad/Happy5 (lower eyelids)
(call_script, "script_set_keys_and_values", 2, 05, 66), #H Sad/Happy6
(call_script, "script_set_keys_and_values", 1, 50, 77), #A' Sad/Happy7
(call_script, "script_set_keys_and_values", 2, 0, 6), #H
(call_script, "script_set_keys_and_values", 1, 0, 7), #A'
(call_script, "script_set_keys_and_values", 2, 0, 6), #H
(call_script, "script_set_keys_and_values", 2, 0, 7), #A'
(call_script, "script_set_keys_and_values", 2, 0, 6), #H
(call_script, "script_set_keys_and_values", 2, 70, 70), #A' Blink0 (open eyelids)
(call_script, "script_set_keys_and_values", 2, 5, 6), #- Sad/Happy6
(call_script, "script_set_keys_and_values", 2, 76543210, 3500000), #XX Sad/Happy5 - smile remains
(assign, "$talk_frame", 1), #start lips
#Sound
(stop_all_sounds, 1),
(play_sound,"snd_evl_from_me"),
]],
Code:
("set_keys_and_values", [
(store_script_param_1, ":step"),
(store_script_param_2, ":keys"),
(store_script_param, ":values", 3),
(assign, ":prev_frame", "$talk_length"), #for blending
(val_add, "$talk_length", ":step"),
(troop_set_slot, "trp_temp_array_a", "$talk_length", ":keys"),
(troop_set_slot, "trp_temp_array_b", "$talk_length", ":values"),
(try_begin), #BLEND MIDDLE FRAME
(ge, "$talk_length", 3),
(eq, ":step", 2),
(store_add, ":blend_frame", ":prev_frame", 1),
(troop_get_slot, ":prev_keys", "trp_temp_array_a", ":prev_frame"),
(neq, ":prev_keys", -1),
(troop_get_slot, ":prev_values", "trp_temp_array_b", ":prev_frame"),
(assign, ":mid_keys", 0),
(assign, ":mid_values", 0),
(assign, ":end", 8),
(try_for_range, ":i", 0, ":end"),
(store_mod, ":prev_key", ":prev_keys", 10), #1305 -> 5, 3, 0, 1 (!do not start with 0 morph key)
(val_div, ":prev_keys", 10), #1305 -> 130,13, 1, 0<- end loop
(store_mod, ":prev_value", ":prev_values", 10),
(val_div, ":prev_values", 10),
(assign, ":endd", 8),
(try_for_range, ":j", 0, ":endd"),
(store_mod, ":key", ":keys", 10),
(val_div, ":keys", 10),
(store_mod, ":value", ":values", 10),
(val_div, ":values", 10),
(try_begin),
(eq, ":key", ":prev_key"), #->Blend them
(val_add, ":value", ":prev_value"),
(val_div, ":value", 2),
(try_begin), #EXCEPTION (0,X) -/-> (0B,XY), BUT (B0,YX)
(ge, ":mid_values", 0),
(eq, ":mid_keys", 0),
(val_mul, ":keys", 10),
(val_add, ":mid_keys", ":key"),
(val_mul, ":values", 10),
(val_add, ":mid_values", ":value"),
(else_try),
(val_mul, ":mid_keys", 10),
(val_add, ":mid_keys", ":key"),
(val_mul, ":mid_values", 10),
(val_add, ":mid_values", ":value"),
(try_end),
(assign, ":endd", 0),
(try_end),
(try_begin),
(eq, ":keys", 0),
(assign, ":endd", 0),
(try_end),
(try_end),
(try_begin),
(eq, ":prev_keys", 0),
(assign, ":end", 0),
(try_end),
(try_end),
(neq, ":mid_values", 0), #":midkeys" can be 0 if just the 0. morph blends
(troop_set_slot, "trp_temp_array_a", ":blend_frame", ":mid_keys"),
(troop_set_slot, "trp_temp_array_b", ":blend_frame", ":mid_values"),
(try_end),
]
),
Code:
face_morph = (
0, 0, 0.04, # reactivate delay 25 frame per sec
[
(ge, "$lipsync_agent", 0),
(try_begin),
(ge, "$talk_frame", 1),
(le, "$talk_frame", "$talk_length"),
(troop_get_slot, ":key_code", "trp_temp_array_a", "$talk_frame"), #example: 135
(try_begin),
(ge, ":key_code", 0), # NOT -1 (0 -> 777777`)
(troop_get_slot, ":value_code", "trp_temp_array_b", "$talk_frame"),
#(assign, reg0, ":key_code"),
#(display_message, "@K:{reg0}"),
#(assign, reg0, ":value_code"),
#(display_message, "@V:{reg0}"),
(agent_get_troop_id, ":lipsync_troop", "$lipsync_agent"), #<- from dialogue
(assign, ":end", 8),
(try_for_range, ":i", 0, ":end"),
(try_begin),
(store_mod, ":key", ":key_code", 10), #1305 -> 5, 3, 0, 1 (!do not start with 0 morph key)
(val_div, ":key_code", 10), #1305 -> 130,13, 1, 0<- end loop
(store_mod, ":value", ":value_code", 10),
(val_div, ":value_code", 10),
(str_clear, s0),
(str_store_troop_face_keys, s0, ":lipsync_troop"),
(face_keys_set_morph_key, s0, ":key", ":value"),
(troop_set_face_keys, ":lipsync_troop", s0),
#(assign, reg0, "$talk_frame"),
#(assign, reg1, ":key"),
#(assign, reg2, ":value"),
#(display_message, "@{reg0} : {reg1} = {reg2}"),
(try_begin),
(eq, ":key_code", 0),
(assign, ":end", 0),
(try_end),
(try_end),
(try_end),
(try_begin), #EXCEPTION : has helm on
(troop_get_inventory_slot, ":item_no", ":lipsync_troop", ek_head),
(ge, ":item_no", 0),
(agent_has_item_equipped, "$lipsync_agent", ":item_no"),
(agent_unequip_item, "$lipsync_agent", ":item_no"),
(agent_equip_item, "$lipsync_agent", ":item_no"),
(else_try),
(agent_equip_item, "$lipsync_agent", "itm_full_helm"),
(agent_unequip_item, "$lipsync_agent", "itm_full_helm"),
(try_end),
(try_end),
(val_add, "$talk_frame", 1), #increment
(else_try),
(ge, "$talk_frame", 1),
(assign, "$talk_frame", 0), # Reset
(assign, "$talk_length", 0),
(assign, "$lipsync_agent", -1),
(try_end),
],
[]
)
Code:
talkress_face_keys = [
(10,0,0.0,0.7, "lip 0"),
(20,0,0.0,0.7, "lip 1"),
(30,0,0.0,0.7, "lip 2"),
(40,0,0.0,0.7, "lip 3"),
(50,0,0.1,0.7, "lip 4"),
(60,0,-1.0,1.0, "Happy/Sad"),
(70,0,-0.3,0.7, "Angry/Wonder"),
(80,0,0.0,1.0, "Blink"),
(90,0,0.0,1.0, "N/A"),
(100,0,0.0,1.0, "N/A"),
(110,0,0.0,1.0, "N/A"),
(120,0,0.0,1.0, "N/A"),
(130,0,0.0,1.0, "N/A"),
(140,0,0.0,1.0, "N/A"),
(150,0,0.0,1.0, "N/A"),
(160,0,0.0,1.0, "N/A"),
(170,0,0.0,1.0, "N/A"),
(180,0,0.0,1.0, "N/A"),
(190,0,0.0,1.0, "N/A"),
(200,0,0.0,1.0, "N/A"),
(210,0,0.0,1.0, "N/A"),
(220,0,0.0,1.0, "N/A"),
(230,0,0.0,1.0, "N/A"),
(240,0,0.0,1.0, "N/A"),
(250,0,0.0,1.0, "N/A"),
(260,0,0.0,1.0, "N/A"),
(270,0,0.0,1.0, "N/A"),
(280,0,0.0,1.0, "Post-Edit"),
]
.
.
.
skins = [
.
.
.
(
"talkress", skf_use_morph_key_10,
"woman_body", "woman_calf_l", "f_handL",
"talkress_head", talkress_face_keys,
["woman_hair_p","woman_hair_n"
.
.
.
]Loading speech sounds
These two lines (module_dialogues.py) add "F" and "R" sound to the playlist.
(call_script, "script_set_keys_and_values", 2, 4, 7), #F
(call_script, "script_set_keys_and_values", 2, 24, 50), #R
2: DELAY - 2 frame steps from the previous sound +(2*40ms or 2*25[1/s])
2: ID#2 (F,V)
4: ID#1 (E,R,K,G,H)
5: value#2 - 5 in [0-7] range: R sound is less "open" than E, a little gap between the teeth is enough (use a mirror)
0: value#1 - 0 in [0-7] range: switches off the previously used shape for F
The second line contains a pair of IDs and values. It zeroes out the previous "F" sound (ID=4, value=0) to shape the mouth to an "R" sound (ID=2, value=5) in 2 frame later.
I use 2 frame steps frequently. That's usually a good tempo, and my code calculates average shapes for the in-between frames to smooth out the animation. When the tempo must be changed (other than 2 frame steps needed) to be in synchron with the audio, you have to make the changes gradually to keep the animation smooth.
IDs and values are construed from right to left (<-), if there are less values than IDs, then the value of the undefined ID will be 0.
IMPORTANT to do NOT start these number lines with 0: You can change the order of IDs (don't forget to change the values too), and skip the starting zeros of values.
For example, the first call - which resets the default morph values -
Code:
(call_script, "script_set_keys_and_values", 1, 76543210, 3300000), #XXTherefore the value of the leftmost id (7) will be set to 0. (Blink=0 -> start dialogue with eyes opened)
You can find more working code samples in WarDrobe ModuleSystem.
Corrections and suggestions are welcome.
Edit:
module_skins.py code sample added
list of sounds order corrected
note inserted about M,B,P sounds
)
