Pioneering a technique to preserve emotional performance in AI-enhanced 3D animation
As part of my comprehensive investigation into AI-enhanced visual media at USC's School of Cinematic Arts, I conducted a critical experiment to address a fundamental challenge: preserving facial expressivity while improving visual quality. After observing how standard AI enhancement flattened emotions in facial performances, I developed a specialized workflow using a highly expressive Overwatch character (DVA) as my test subject, isolating facial processing from body rendering.
Working in early 2023, when generative AI was still in its infancy and dedicated video models didn't exist, I crafted innovative solutions that pushed the boundaries of what was then possible with image-based AI models. This experiment established key techniques that directly informed my approach to recreating scenes from Ex Machina and later character-focused projects, establishing a foundation for preserving performance quality in AI-enhanced rendering.
Enhance the realism of 3D animation while preserving the emotional fidelity of facial performances
April 2023, during the early days of AI-enhanced rendering techniques
Lead Researcher & Developer, designing and implementing the entire workflow
Stable Diffusion 1.5, Auto1111 WebUI, ControlNet, MediaPipe Face Detection, After Effects, DaVinci Resolve, Topaz Video AI
During my exploration of AI rendering techniques, I encountered a significant obstacle: while AI enhancement provided impressive visual improvement to 3D animations, it came at the cost of emotional fidelity. Facial expressions became rigid and emotionless, defeating the purpose of performance capture and reducing the narrative impact.
I developed a novel dual-processing approach where I would process faces and bodies separately with different parameter sets, then composite them back together. This allowed me to optimize the parameters for facial expressivity without compromising the quality of the rest of the render.
Before & After: DVA Facial Expressivity Test - Note the preserved emotion and facial detail
I approached this challenge methodically, breaking down the problem into discrete steps that could be solved individually before being integrated into a cohesive workflow.
I conducted systematic testing to understand exactly how facial expressivity was being lost in AI rendering. Through careful parameter testing, I discovered that the denoising strength needed for photorealism was too aggressive for preserving subtle facial movements, and that the AI prioritized overall frame coherence over local detail preservation.
I developed a specialized process for generating accurate face masks from each frame. Using MediaPipe Face Detection with specific parameters, I created masks that precisely isolated facial areas for targeted processing.
Face Detection Settings: - Detector: Normal (OpenCV + FaceMesh) - Minimum face size in pixels: 30 - Rotation threshold: 20 - Single mask per image: Enabled
I conducted extensive experimentation to find the optimal parameters for facial processing. The key discovery was that reducing the denoising strength to 0.5 (from 0.65 used for bodies) preserved significantly more expressivity while still enhancing visual quality.
Inpaint Settings for Faces: - Mask mode: Inpaint masked - Masked content: Original - Inpaint area: Only masked - Only masked padding: 32px - Sampling method: Euler - Sampling steps: 30 - CFG Scale: 8 - Denoising strength: 0.5 ControlNet Settings: - Preprocessor: mediapipe_face - Model: control_v1p_sd15_mediapipe_face - Weight: 1 - Guidance: Full range (0-1)
The final stage involved tracking and compositing the enhanced faces onto the body renders. This required precise alignment and seamless blending to ensure natural integration. I further refined the process with frame interpolation techniques to improve smoothness and reduce flicker.
As I noted in my journal (April 17, 2023): "I need to try it and include it in the workflow... imagine: my current method plus three controlnets, one for body, one for depth/bg, and one for faces. All processing for each batched image."
This experiment yielded remarkable results that not only solved the immediate problem but established foundational techniques that would inform my future AI rendering work.
Beyond the technical metrics, this experiment achieved several significant outcomes:
"The preservation of subtle facial movements while still achieving photorealistic enhancement demonstrates a sophisticated understanding of both AI models and character animation requirements."
— Faculty assessment of the DVA facial expression technique
This project represented a critical juncture in my exploration of AI rendering techniques, yielding insights that would shape my approach to subsequent work.
This project taught me the importance of problem decomposition in AI—breaking complex challenges into specialized tasks. Working at the cutting edge of AI in early 2023, with minimal documentation and precedent, forced me to develop innovative approaches and trust my experimental results. The recognition from professors and industry professionals validated my unconventional approach and encouraged me to continue pushing boundaries in subsequent projects.
