Pioneering the first AI rendering workflow that transforms basic 3D animation into photorealistic cinematic sequences
I took on the challenge of recreating a famous scene from Ex Machina using Unity game engine, performance capture, and face capture. Dissatisfied with the initial results, I ventured into the uncharted territory of AI to enhance the video's realism. Despite the scarce resources in early generative AI (early 2023), I crafted a new AI-powered rendering workflow that dramatically improved quality, making scenes look incredibly lifelike.
What started as an experiment led to unexpected breakthroughs: my AI approach didn't just mimic the movie's style; it learned physics for better lighting and even simulated realistic muscle movements all on its own. Achieved with Stable Diffusion, this journey required persistence, extensive learning, and deep technical knowledge. The resulting workflow established a revolutionary new approach to enhancing 3D animation with AI, creating a rendering pipeline that could transform basic 3D animation into cinema-quality visuals.
Create the first-ever AI rendering workflow for transforming basic 3D animation into photorealistic footage
8 months of development (October 2022 - May 2023) during the early days of generative AI
Creator, Researcher, Developer, and Technical Director for the entire workflow
Unity, Motion Capture, Facial Capture, Stable Diffusion, Auto1111, ControlNet, After Effects, DaVinci Resolve, Topaz
The core challenge was bridging the significant gap between game engine renders and cinematic photorealism, with numerous obstacles along the way:
I developed a comprehensive, multi-stage workflow that addressed each challenge systematically:
Comparison: Original movie vs Unity scene before AI enhancement
Early test showing the dramatic quality improvement with AI enhancement
This project involved a complex, multi-stage development process that evolved significantly as I discovered new techniques and overcame challenges.
The project began with capturing performances at USC's mocap lab. As I recorded in my journal (January 27, 2023): "By 5:30 went to mocap lab with Aria. We decided to do ExMachina ending scene and join forces with another group. Aria tackled me and then we struggled on the ground and then I got 'stabbed' by the guy on group 2 acting as the second android. All in mocap suits. It was so fun."
This initial stage included:
Left: Performance capture in mocap suit; Right: Scene implementation in Unity
Dissatisfied with the Unity output quality, I began exploring AI enhancement. My journal from February 26, 2023, captures this turning point: "Watched a Corridor Digital video where they cracked the code of making video look like traced anime. Then went home." This inspired me to try the reverse process - using AI to make CG look more photorealistic.
Key developments during this phase:
Early test showing promising results but significant flickering issues
The initial tests showed promising quality improvements but suffered from severe frame-to-frame inconsistency. I developed a sophisticated approach to address this issue, as noted in my journal (March 23, 2023): "Modified my workflow by adding warp stabilization on AE and passing through topaz sharpen. Conclusion: different order may be better. Start with topaz video 60fps and end with that too somehow."
This phase involved:
Improved quality with reduced flickering, but facial expressivity and character likeness issues remained
To accurately capture the likeness of the original actors, I developed specialized LoRA models. This required extensive dataset creation and training experimentation, as I noted in my journal (April 2, 2023): "I spent the day on finishing the experiment."
This stage included:
Character creation process for Nathan (left) and Ava (right)
To maintain emotional fidelity, I developed a technique that processed faces separately from bodies. My journal from April 18, 2023, captures this: "Recorded the facial expressions... experimented a bunch with controlnet and understood it... tried controlnet with current video and it sorta worked."
Key developments:
Face ADR recording to enhance facial animation quality
The final phase involved bringing all components together into a cohesive workflow. As I noted in my journal on April 25, 2023: "Finished the exmachina test. Showed it to John and Emre. Emre said he was satisfied. John and I had a long conversation... He insisted I make a blog about the process."
This stage included:
Final result: AI-enhanced recreation of Ex Machina scene
The final workflow I developed consisted of multiple stages carefully orchestrated to achieve optimal results:
Through extensive experimentation, I identified the optimal Auto1111 parameters for photorealistic rendering:
Auto1111 Workflow: 1. Navigate to `img2img` tab 2. Upload Unity render as source image 3. Use models: `realisticVisionV20_v20` or `CinematicDiffusion_v1` 4. Configure detailed prompts incorporating: - Character descriptors (e.g., "Ava Machina") - Style targets - LoRA references (detail, style, corrective like `badhands`) - Negative embeddings (e.g., `NG_DeepNegative_V1_75T`) 5. Set img2img parameters: - Resize Mode: Crop and resize - Sampling Method: Euler or Euler a - Sampling Steps: 25 - Resolution: ~910x512 (aspect ratio dependent) - CFG Scale: 7-8 - Denoising Strength: 0.63-0.65 (critical range) 6. Configure ControlNet: - Enable OpenPose for body structure - Enable Canny for edge preservation - Enable MediaPipe Face for facial details - Set weight typically to 1, with full guidance range 7. Activate script: `img2img alternative test` - Decode CFG scale: 1 - Decode steps: 25 - Randomness: 0 - Sigma adjustment: True 8. Execute and evaluate results 9. Use inpainting for targeted fixes as needed
One of the most fascinating aspects of this project was discovering the AI's emergent capabilities - things it could do without explicit training:
The final result was a breakthrough in AI-powered rendering, creating a complete workflow that could transform basic 3D animation into photorealistic footage. This project demonstrated the potential of AI as a rendering tool, particularly in its early stages before dedicated video models were widely available.
Beyond the technical metrics, this project achieved several groundbreaking outcomes:
"For the time, this was bleeding edge. I had not seen anything like this before."
— CEO of ComfyUI, commenting on the project
This project represented a significant milestone in my exploration of AI for creative applications, yielding insights that have shaped my approach to technology and problem-solving.
This project fundamentally changed my understanding of AI's potential in creative fields. It demonstrated that even at its early stages, AI could serve as more than just a content generator – it could be a sophisticated rendering tool that understands physics, lighting, and anatomical details without explicit programming. The journey from a simple Unity scene to a photorealistic film recreation proved that determination and methodical experimentation can overcome technical limitations. What began as a technical experiment evolved into a vision for democratizing high-end visual production capabilities.
In the midst of developing this project, I experienced a pivotal moment during a set visit to James Cameron's Avatar production. Having deep familiarity with the cutting-edge technology being used on Avatar, I immediately recognized the parallels between their advanced performance capture systems and my own experimental AI approaches. As I documented in my journal (March 3, 2023): "The Avatar visit was incredible. Seeing their performance capture setup, face rigs, and especially their real-time previs rendering made everything click. Their setup costs millions, but I'm doing something conceptually similar with just my laptop and AI. I see a very palpable path... using my new AI workflow idea in one of my projects... What's next? AI workflow." This experience was transformative – standing at the intersection of Hollywood's most advanced production technology and my own AI innovation. While the Avatar team achieved photorealism through massive computing infrastructure costing millions, I was pursuing a parallel goal that could run on a standard laptop. My hope is that someday this research will evolve into technologies that bring Avatar-level production capabilities to independent creators and smaller studios – making photorealistic rendering available to anyone with a creative vision.
My early Avatar-inspired experiment using the AI rendering workflow I developed
The group of talented creators being taught by masters of performance capture technology at the Avatar set visit
