Why Does the Deepfake Version of Luke Skywalker in The Mandalorian Look Better Than the CGI One?
Why Does the Deepfake Version of Luke Skywalker in The Mandalorian Look Better Than the CGI One?
The Realism of Deepfake Technology
Deepfake technology has revolutionized the way we approach visual storytelling, particularly in movies where the recreation of beloved characters is crucial. Deepfake technology uses machine learning algorithms to analyze and replicate facial movements and expressions, which can result in more realistic and nuanced facial animations compared to traditional CGI. This is why the deepfake version of Luke Skywalker in The Mandalorian looks better than the CGI version used in the episode.
Deepfake technology relies on extensive footage of the original actor, allowing for a more accurate representation of their likeness and performance. In contrast, CGI often relies on a combination of models and textures that might not fully capture the actor's specific features or expressions. This difference in data source is reflected in the level of naturalness and realism that the deepfake version can achieve.
Data Source
Deepfakes can be trained on extensive footage of the original actor, allowing for a more accurate representation of their likeness and performance. In contrast, CGI often relies on a combination of models and textures that might not fully capture the actor's specific features or expressions. For example, in the fan-made deepfake version of Luke Skywalker, the creators could have worked with ample footage of Mark Hamill, ensuring that the facial expressions and movements are more nuanced and realistic.
Lighting and Integration
A well-executed deepfake can blend more seamlessly with the live-action footage, especially if it is matched closely to the lighting conditions of the scene. This is particularly important when a character is fighting bad guys, as lighting and texture are crucial for convincing visual effects. In the case of Luke Skywalker, the deepfake version matched closely with the lighting and texture of the surrounding environment, making it appear even more lifelike.
Audience Perception
Finally, audience perception plays a significant role. Viewers may have a bias towards the deepfake because it evokes nostalgia by using the original actor's likeness and performance. This emotional connection can enhance the perception of realism, even if the deepfake version is slightly more polished than the CGI version. The deepfake version of Luke Skywalker in The Mandalorian resonated more effectively with audiences because of this emotional connection.
The Evolution of Deepfake vs. CGI
The deepfake version of Luke Skywalker is a testament to the superior capabilities of deepfake technology. By definition, deepfakes are videos in which a person's face and/or voice are replaced with someone else's likeness through an AI. The underlying technology and machine learning processes have evolved significantly since the early 90s, with the term "deepfake" combining the words "deep learning" and "fake."
The process of creating a deepfake involves using machine learning algorithms, typically involving an autoencoder and a generative adversarial network (GAN). The autoencoder is a computer's way of seeing a face and determining all the possible ways it can appear. The GAN then pits two AI algorithms against each other: one creating forgeries and the other grading its efforts, teaching the synthesis engine to make better forgeries. This, combined with the extensive data source, makes deepfakes more realistic than traditional CGI.
On the other hand, CGI has been revolutionary, but the software and technology of deepfake are much newer and do a superior job of blending seamlessly into the action. While CGI is still catching up, it would make more sense for studios to use this technology. It is cheaper, simpler, and more convincing.
The Mandalorian’s CGI vs. Deepfake
In the The Mandalorian Season 2 finale, the de-aging CGI of Luke Skywalker falls short of the high standards set across the rest of the show. The deepfake version, on the other hand, looks almost like he did in 1983, with more texture in his face and more alive eyes. Even Mark Hamill, who portrays Luke Skywalker, might find it weird to see a young version of himself, even if it looks like he's just been to the dentist.
The Techniques Behind Deepfakes
Deepfakes use deep learning technology, a branch of machine learning that applies neural net simulations to massive data sets to create a fake. AI effectively learns what a source face looks like at different angles to transpose the face onto a target, as if it were a mask. This is achieved using generative adversarial networks (GANs), which pit two AI algorithms against each other, one creating forgeries and the other grading its efforts.
The default technique of bringing back a deceased actor, such as Carrie Fisher, involved complex and expensive pipelines. However, a nerd with some deepfake software can replicate this process in a single day, emphasizing the efficiency and realism of deepfakes. The de-aging of Mark Hamill in The Mandalorian is a prime example of this technique, but it falls short in terms of realism compared to the fan-made deepfake.
Challenges and Ethical Considerations
The question remains whether Hollywood directors will fully adopt deepfake technology on a massive scale. The massive render time and required supercomputing power pose significant challenges for traditional studios. The amount of render time for even a short scene would take a huge amount of time with the most powerful supercomputers.
Moreover, the legal framework surrounding deepfakes is still evolving. While contemporary deepfakes are constructed by AI, impersonating people, the "drunk Pelosi" video is often cited as an example of confusion. This video slowed down and pitch-corrected a video of Nancy Pelosi, giving her a slurred-speech effect. However, it is important to note the massive legal caveat: what if the actor wants to protect their intellectual property and legally prevent the industry from using their face in AI-generated movies?
As with any new technology, there is confusion and ethical concerns, but the potential for deepfakes to enhance visual storytelling and reduce costs makes them a compelling alternative to traditional CGI. As technology continues to evolve, we may see a significant shift in the film industry.