Lip sync setup & animation tutorial

Objectives:

To prepare your 'Actor' to perform lip sync to pre-recorded dialogue

Softimage Tools Used:

• Get Primitive Null
• Draw Extract
• Tag Set Cluster
• Tag Cluster List
• Constraint Object To Cluster
• Skeleton Group As Skeleton
• Skin Global Envelope
• Shape Shape Interp Weighted
• Shape Shape List Additive
• Skin Envelope Assignment Modify Weights
• Savekey Object Shape
• F-Curve Select Object Shape Weights

This tutorial assumes that you have constructed the head of your character using the Radial Modelling Method. There are a number of Radial Modelling Methods online if you are unfamiliar with this technique. Perhaps the best tutorial online can be found at WWW.3DRENDER.COM. The Vault also possesses a radial modelling tutorial supplied by Softimage from the Technique section of their web site found at WWW.SOFTIMAGE.COM.

MOUTH SETUP

Extracting The Mouth Spline, & Setting Clusters with Centers

1.) You should already have constructed the head of your character in a neutral, open-mouthed position using the radial modelling method. This default mouth position will allow the envelope weighting to be evenly distributed over the lips control Nulls. A NURBS head will work most efficiently due to the added precision in extracting splines from a surface, however for our purposes, it is best to use a B-spline Patch due to hardware limitations.

2.) Click on the Preferences button, and ensure that the Create Modelling Relations options has been deactivated. If this button is not deactivated, your character will not be able to perform any lip sync after it has been setup.

3.) Select the head in Single mode, and use the Draw Extract Command in MODEL (F1) to extract a spline from around the lip area.

This extracted spline will be shape animated to drive the lip movement of your head.

4.) Select the extracted spline in single mode, and tag a single point on the curve. Click on Tag Set Cluster to bring up the Set Cluster window.
Click in the white box to the left of Create Center, and then click the NEW button.

You have just created a cluster complete with its very own center. To verify this, you should see that a null has been created in the same space as the point you first tagged. This null is the cluster's center.

Select the Spline, and untag the point you first tagged.

5.) Repeat the above step for each successive point along the spline until each point has become a cluster with its own center. When this step has been completed, you will have as many nulls as there are points on your spline.

In Multi mode, you can now select these nulls and hide them using Display Hide Toggle Desel. Hidden. These Nulls are simply centers, and will only serve to confuse you during the next step.

Null Skeletons & Cluster Constraints

6.) Get Primitive Null. Duplicate this null so that you end up with as many Nulls as there are points on your extracted 'lip' spline. You may want to move each duplicated Null so that they are not all stacked on top of one another. The less confusion you have to deal with the better! These Nulls will become the skeletons that will drive the lip animation for your characters head.

7.) Select your extracted spline. In MOTION (F2 is the hotkey) click on Shape Cluster List to bring up the Cluster List window. Within the Cluster List window, you will see the names of all of the clusters that you have created in steps 4 & 5. Click once on the first cluster (Cluster0), and then click on OK. You will have exited the Cluster List window, and the respective cluster on your spline will now be highlighted.

8.) Select one of the nulls that you created in step 6. It doesn't matter which one.

In the MOTION module (F2) click on Constraint Object To Cluster. If you look down at the mouse line at the bottom of your screen, you will see that it prompts you to 'pick a reference' using either the Left or Middle mouse button. Since the cluster has been activated on the extracted spline, click on the spline as a reference. The null will be constrained to the Cluster that you highlighted in step 7.

9.) Repeat step 7 once again, The first cluster already has a Null constrained to it, so its purpose has been served. Select the next cluster in your cluster list. Click OK. The next cluster (Cluster1) will now be highlighted on the extracted spline.

Select the next available free Null and click on Constraint Object To Cluster again. (Note; Softimage remembers the last command used on an object. If you look down at the mouse line when you select the new Null, you will notice that 'Pick Reference' is still activated) With the new Null selected, click on the spline. This null will snap to the next cluster that you activated in the Cluster List window.

Repeat this step until all of the Nulls have been constrained to a cluster on the 'lip' spline.

Depending on the size of your characters head in Softimage Units, the Nulls that you have just constrained to the clusters may be too large. You can select each of the Nulls and scale them up or down to suit the relative sizes of each of your characters.

Transforming Nulls Into Skeletons & Enveloping Nulls to the Head

10.) The setup of the character's mouth is now ready to be turned into a skeleton and enveloped in to the head.

Select one of the Nulls. In your ACTOR (F3) module, choose the command Skeleton Group As Skeleton. Your mouse line will prompt you to 'Set Skeleton'. Click on the Null that you have selected. It will turn Green. If you frame it in your Schematic viewport, you will notice that it has the word 'Skel' written in red above it, indicating that the null now has the same physical properties as a 2D or 3D chain of Joints.

Select the next null and click once on it to turn it into a skeleton object (the Set Skeleton prompt is still activated along your mouse line).

Repeat until all of the constrained Nulls along the spline have been turned into skeletons.

11.) Select your characters head in single mode, and click on Skin Global Envelope, and pick one of the nulls. Accept the default switches in the Envelopes Assignment Box, unless you prefer Full weighting as opposed to weighting for an enveloping option.

Select the head again and envelope in the rest of the nulls.

When you have finished the enveloping, select the spline, and parent it to the character's head as a child. This will allow the lips to move along with the head when the character is performing any actions.

Your characters head has now been 'boned', all that is left for us to do before beginning animation, is the weighting of the points on your characters head.

12.) Tag points on your characters head, and using the Skin Envelope Assignment Modify Weights command, assign and weight the points around your characters mouth to the various Nulls that make up its skeleton.

It will save you a great deal of time if you name your Nulls using the Info Selection command. Names such as "TLLIP (top left lip) BRLIP (bottom right lip) etc. will make it much clearer when choosing which points to weight to which nulls.

This step requires the most patience; after all, careful weighting will determine how well the characters mouth area deforms. There is nothing worse than animating a character whose geometry 'crashes' through itself. It is also very distracting to a prospective employer or an audience viewing your work.

More on enveloping techniques in another tutorial.

ANIMATING YOUR MODEL

We have covered a lot of ground up to now, the benefits of this setup method are ready to be harnessed in the precise control that you now have over your lip animations.

Essentially what you are going to do is to create your own set of 3D sliders, and add expressions to the sliders (Nulls) which will associate them to a series of phoneme lip shapes that you have sculpted.

It will be handy for the purposes of the next few steps if you have a pictorial reference of what the mouth looks like when speaking certain phonemes. The Preston Blair books have an excellent guide to follow along with.

13.) Switch to your MOTION (F2) module, and click on Shape Shape Interp. Weighted.

Weighted interpolation creates one F-Curve for each key shape. The term weighted, comes from the fact that you can adjust each shapes function curve to emphasize any key shape or combination of key shapes.

Linear interpolation means that the physical points of the deformed object will travel in a linear direction.

You all should be in the practice of starting your animations after frame 100. We are now going to find a use for frames 1-99. Since we are going to be creating numerous lip phoneme shapes, and lip expressions, we can use the each frame from 1 - 99 to store a shape. Potentially, we could store 99 lip shapes, although you won't need nearly that many.

Begin at frame 1. Select your lip spline in single mode, and click on Savekey Object Shape.

Move the timeline to frame 2, and move the points on the spline to create the next shape. Click on Savekey Object Shape.

Continue on until you have created all lip shapes that you will need. Remember, each lip shape should be stored on a different frame.

14.) Click on Shape Shape List. In the window that pops up, you will see a box that contains all of the names of the shapes you have created. To the right of that there are two options Average and Additive. Make sure that Additive is selected. Additive will take the information from two shapes and add their properties together instead of averaging the two.
Click OK.

The result, a wider range of emotion available to be animated in your characters face.

To check and see if everything is working well, scroll through the frames that have a keyshape saved on them, all the while looking at your characters mouth. As each frame goes by, it will assume the shape saved on respective frames.

Bring in as many NULLS using Get Primitive Null as you have KEY SHAPES, and arrange them side by side in your right view window on the origin. To make matters easier, you can create a border around your nulls using a primitive square. It is a good idea to color it, so the null is easily spotted in your schematic window. This way, you can frame your slider nulls by pressing the "F" hotkey when the null is selected.

Select each of the nulls in single mode and rename them using Info Selection. Think of names that correspond to each phoneme lip shape that you have sculpted, i.e. Ooh, Ff, Th etc.

Shape Weights & Expressions

This next step involves associating nulls to each shape that has been saved on the lip spline. Through the use of simple Expressions, we will create a slider system similar to a mixing board that will allow you to animate the various phoneme shapes you have created for your character.

15.) Select the lip spline in single mode. Press F2 to switch to the motion module. In the lower right of your screen, click on Expressions Edit. The Expressions editor window will pop up. Expressions are simply a string of characters (numbers and letters) that define relationships between various elements in a scene.

For example; A = B, is a simple expression showing that A is equal or constrained to B. Expressions can be used for to create relationships for any parameter that can be animated.

You will notice that the lip spline's name appears in the affected element box. The lip spline you will remember, has all of the shapes saved on it. When we associate the slider Nulls to each shape on the spline, it is the animation of the Nulls that are going to affect how the shapes interact with each other to create convincing lip sync.

Thus, since the Nulls drive the shapes, the lip spline is the Affected Element!

An easy analogy would be that of a CD and a CD Player. A Compact Disc would be considered the affected element, because when placed into the tray of a CD player, it is going to rely on the CD players drive belt to rotate it.

IT IS AFFECTED DIRECTLY BY THE DRIVE BELTS ROTATION. THEREFORE PUT INTO SOFTIMAGE LANGUAGE AN EXPRESSION TO DESCRIBE THIS INTERACTION MIGHT LOOK LIKE THE FOLLOWING:

CD:RotY=DRIVE BELT.RotY

16.) Make sure that the cursor is positioned after the name of in the Affected Elements box. In the middle of the window is a box that reads F-Curves. Click on it. A window will pop up that contains all of the F-Curves possible for your Affected Element (Lip Spline). Your lips Shape Interpolation has been set to Weighted in step 13. so their will be a number of F-Curves in the list named Wshp0, Wshp1, Wshp2.

Each Wshp is equivalent to one of the keyshapes saved on the spline. Since you saved your shapes from frame 1 to X, the shape weight F-Curves will roughly correspond to each frame. Select the first shape weight F-Curve, and press OK.

The Wshp0 F-curve is inserted into the Affected element line

You should now have the following in your Affected Element line

LipSpline.Wshp0 (LipSpline being the name of your model)

Now click in the box below marked Expression. This is where you will associate the Lip Splines Shape Weight to the respective slider Null that describes the Phoneme shape.

Click in the Scn Element (Scene Element) box, and in the window that pops up, select the name of the null that corresponds to the Shape Weight for the Lip Spline that you have selected in your Affected Elements box.

For example; if Wshp0 is an "Oooh" phoneme shape, and LipSpline.Wshp0 is the affected element, then select the Scene Element "Oooh" as the first object on your Expressions line.

Press OK

Click in the F-curve box to bring up all possible F-curves for the Slider Null. The F-Curve that we are looking for is Etrny, (or the Nulls Explicit Translation along the Y- axis) Select Etrny, and click OK.

This associates the Lip Splines Wshp0' to the Y-translation of the Null.

Shape weights F-Curves have one important feature that makes them ideal for use with a mixing board / slider approach. If you call up the F-Curve for a shape weight, you will notice that it runs in value from 0 to 1. A shape weight with a value of zero is off, or has no weight. A shape weight of one is equal to the original shape sculpted, and is thus fully weighted.

A shape weight of 0.5 then, would result in a shape that is halfway between on and off. If you had a shape weight F-curve for a smile, a weight of zero would not have any effect on the model, a weight of one, would yield the smile, and a weight of 0.5 would give a neutral emotion.

The expression still isn't finished. You should have the following element and F-curve in the Expression box:

Null.Etrny (where Null is the name of the phoneme corresponding to your shape.)

The final Expression should read:

(Null.Etrny ) / 10

The brackets are important here, because they tell SOFTIMAGE to perform the math within the brackets first, before dividing the result by 10.

Here is the reasoning for the math;

As you already know, Shape Weight F-Curves run from a value of 0 to 1, 0 meaning that the particular shape is turned off, 1 meaning that the shape is turned on. The nulls that we created in the right view window are arranged on the origin if you remember, so their Y value is 0, which is equivalent to a shape weight being turned off. Dividing the expression by 10 will yield a maximum value of 1 for the null.

Therefore, if you animate the Nulls Y translation from 0 to 1, it will correspond to the shape weights position from 0 to 1.

Repeat step 16 until all of the shape weights and nulls have been accounted for.

17.) Your character is now ready to be animated. By moving the Null sliders, and saving Explicit Translation Keyframes, you can easily, and efficiently animate the lips to pre-recorded dialogue that has been doped out.

Keep in mind that using Shape weights allows you to mix the weight of more than one shape at once, and also allows for you to jump back and forth between any shape that has been keyed.

It takes time and patience to get this method of animation down pat, but your result will most certainly be worth it.

January 30, 1998

Copyright Adam Sale 1998

-Tutorial created by Adam Sale