First Day: AI Anime Stylization

Project Overview

As Motion Capture Operator and AI Supervisor for USC School of Cinematic Arts' first-ever virtual production thesis project, I brought cutting-edge AI stylization techniques to an ambitious Evangelion-inspired narrative featuring mecha combat sequences.

Building on my previous successes with AI rendering workflows in projects like Ex Machina AI Rendering and Immersion: Real Meets AI, I adapted my expertise to a new challenge: transforming CG mecha footage into anime aesthetics. While my previous projects had focused on achieving photorealism from CG inputs, this project required the opposite approach—creating stylized anime visuals from realistic 3D renders.

Working with motion capture data from performers in a complex production environment, I created stylization tests that demonstrated the potential of AI to transform standard CG output into compelling anime-style sequences. My work showcased innovations in handling non-human forms, preserving story-critical elements, and achieving stable frame-to-frame consistency—all significant challenges in late 2023 when this technology was still in its infancy.

🎯

Goal

Transform CG mecha footage from USC's virtual production into convincing anime-style visuals while preserving performance integrity.

⏱️

Timeline

January 2024 - June 2024, working with USC School of Cinematic Arts' thesis production team.

🧠

Role

Motion Capture Operator and AI Supervisor, responsible for mocap testing and creating AI stylization previews.

🛠️

Tools & Technologies

ComfyUI, Stable Diffusion, After Effects, DaVinci Resolve, Giant (mocap software), Motion Capture hardware.

Challenge & Solution

The Challenge

The project presented several significant technical challenges at the intersection of motion capture, virtual production, and AI stylization:

Non-Human Forms: The mecha designs featured unusual shapes and proportions that confused standard AI systems trained primarily on human forms and real-world objects.
Story-Critical Elements: Cockpit screens contained important narrative information that needed to be preserved through the stylization process.
Performance Integration: The project involved capturing performances from multiple actors simultaneously, requiring careful coordination between physical performance and visual stylization.
Temporal Stability: Early 2024 AI video generation still suffered from significant frame-to-frame inconsistency, creating distracting flicker effects.
Technical Complexity: The Giant motion capture software presented a steep learning curve and integration challenges with the production pipeline.

The Solution

I developed a comprehensive approach that addressed each challenge through innovative technical solutions:

Outline and Texture Focus: Rather than relying on standard object recognition, I adapted my AI workflow to emphasize outlines and metal texturization, helping the system recognize and stylize the mechas despite their non-human shapes.
Selective Masking: I implemented a manual masking and compositing workflow in After Effects to preserve cockpit screens and other story-critical elements while allowing surrounding elements to be stylized.
Workflow Adaptation: Building on my experience from the Immersion project, I refined my ComfyUI setup to achieve anime-style results rather than photorealism, demonstrating versatility in my technical approach.
Flicker Reduction: I employed advanced deflickering techniques and optimized processing parameters to achieve maximum stability possible at that time.
Process Refinement: I developed separate processing approaches for mecha sequences versus cockpit scenes, optimizing each for their specific visual requirements.

Process & Methodology

My approach to this project involved a systematic adaptation of established AI workflows to the unique requirements of anime stylization. The process evolved through careful testing and refinement, building on my previous technical explorations while advancing into new aesthetic territory.

1

Motion Capture & Virtual Production Setup

The project began with extensive motion capture sessions at USC's facility. While I had initial experience with the Giant motion capture system, the production ultimately used Motiv for recording performances. This phase involved coordinating multiple actors simultaneously to capture both mecha combat sequences and cockpit interactions.

2

ComfyUI Workflow Adaptation

Building on my previous work with AI stylization in projects like "Immersion: Real Meets AI," I adapted my established ComfyUI workflow for anime aesthetics. This required significant adjustment of models, reference images, and processing parameters to transition from my previous focus on photorealism to a distinctly anime style. The workflow maintained my core approach to video processing while implementing specialized techniques for outline emphasis and metal texturization.

3

Mecha Stylization Challenges

The non-human shapes of the mecha designs presented significant challenges for the AI system, which often attempted to interpret mechanical elements as human features. Through systematic experimentation, I developed a specialized approach that emphasized outlines and metallic textures while suppressing the system's tendency to anthropomorphize mechanical forms. This required careful prompt engineering and parameter adjustment to maintain the mecha's core design while achieving the desired anime aesthetic.

4

Cockpit Scene Processing

The cockpit scenes presented a different set of challenges from the mecha sequences. These scenes contained critical story elements on screens and displays that needed to be preserved through the stylization process. I developed a specialized masking technique in After Effects to protect these elements while allowing the surrounding environment to be fully stylized. This approach required precise rotoscoping and compositing to achieve seamless integration.

Cockpit scene stylization with preserved screen content

5

Flicker Reduction & Output Refinement

A persistent challenge in early 2024 AI video processing was frame-to-frame inconsistency leading to flicker. I implemented a multi-stage deflickering approach that included both pre-processing of source footage and post-processing of AI-generated output. This involved optimization in DaVinci Resolve and specialized compositing techniques in After Effects, pushing the stability limits of what was technically possible at the time.

Technical Deep Dive

This project represented a significant technical advancement in my AI stylization work, particularly in adapting existing workflows to new aesthetic goals while solving novel challenges related to non-human forms.

AI Workflow Evolution

My approach to this project built directly on the techniques I developed for "Immersion: Real Meets AI," but with a crucial difference: instead of transforming CG into photorealism, I was now transforming CG into anime style. This required:

Model Selection: Utilizing anime-specialized Stable Diffusion models rather than photorealistic ones
Reference Engineering: Creating specific anime style reference points for the system to emulate
Parameter Adjustment: Fine-tuning denoising strength, CFG scale, and other parameters for stylized rather than realistic output
Prompt Development: Crafting specialized prompts that emphasized line work, flat shading, and other anime characteristics

Mecha-Specific Challenges

The project's focus on mechas presented unique technical hurdles that required creative solutions:

Shape Recognition: Early 2024 AI systems struggled with non-human mechanical forms, often attempting to interpret them as faces or bodies
Outline Preservation: I developed a technique that prioritized edge detection and outline preservation to maintain the mecha silhouettes
Metal Texturization: Special attention was given to preserving and enhancing metallic surface qualities while transitioning to anime aesthetics
Light Source Handling: The central core light sources in the mechas required specific masking and processing to prevent the AI from misinterpreting them as eyes or other anthropomorphic features

Compositing Workflow

A crucial aspect of this project was the integration of AI-processed elements with story-critical content:

Screen Content Preservation: Manual masking in After Effects allowed for selective stylization that protected important screen content
Multi-Pass Processing: Different elements were processed separately with tailored parameters, then recombined in compositing
Temporal Consistency: Advanced deflickering techniques were applied in DaVinci Resolve to maximize stability
Final Integration: Careful color grading and element blending created cohesive final output that maintained consistent aesthetic quality

Results & Impact

This project demonstrated my ability to apply cutting-edge AI techniques to transform standard CG animation into compelling anime-style visuals, showcasing my early mastery of stylization techniques at a time when the technology was still in its infancy.

Technical Achievements

Beyond the measurable metrics, this project represented several significant qualitative achievements:

Workflow Versatility: Demonstrated the adaptability of my AI processing approach to completely different aesthetic goals.
Non-Human Form Processing: Developed novel techniques for handling complex mechanical forms that confused standard AI systems.
Selective Stylization: Created an effective approach for preserving critical content while stylizing surrounding elements.
Early Technical Leadership: Showcased cutting-edge capabilities at a time when this technology was still emerging and poorly documented.

Mecha stylization test showing before and after transformation from CG to anime aesthetic

Reflection & Learnings

This project represented an important evolution in my exploration of AI for creative applications, yielding both technical insights and valuable project experience.

What Worked Well

Technique Adaptation: Successfully translated my existing workflow to a completely different aesthetic goal.
Problem-Solving Approach: Developed effective solutions for handling non-human forms and preserving critical content.
Stability Optimization: Achieved excellent frame-to-frame consistency despite the technical limitations of early 2024 AI video processing.

Challenges & Solutions

Technology Limitations: Worked at the bleeding edge of AI capabilities, pushing the limits of what was possible at the time.
Expectation Management: Navigated the gap between technical possibilities and collaborator expectations during a period of rapid technological evolution.
Technical Complexity: Balanced sophisticated processing techniques with the practical realities of production timelines and available computing resources.

Future Applications

Style Adaptation: The techniques developed could be applied to transform footage between virtually any visual styles.
Production Integration: The compositing techniques could be incorporated into standard production pipelines to achieve stylized effects on a larger scale.
Educational Value: The project offers valuable insights into the challenges and solutions for AI stylization of non-human forms and specialized content.

Personal Takeaway

This project reinforced my position at the forefront of AI-enhanced visual production, demonstrating my ability to adapt established techniques to new aesthetic challenges. Working with the USC School of Cinematic Arts' production team provided valuable experience in applying cutting-edge technology in a collaborative academic environment, while testing the boundaries of what was technically possible with early 2024 AI capabilities. The project represented a natural evolution of my technical approach, showcasing versatility in adapting my workflows to diverse visual styles.