New document published by Disney Research in partnership with ETH Zurich describes a fully automated neural network-based method for swapping faces in photos and videos – the first such method that results in high end results resolution and megapixel resolution according to the researchers. This could make it suitable for use in film and television, where high resolution results are essential to ensure that the end product is good enough to reliably convince viewers of their reality.
The researchers specifically intend to use this technology to replace the performance of an existing actor with the face of a substitute actor, for example during the aging or increasing age of a person, or potentially during the representation of a deceased actor. They also suggest that it could be used to replace the faces of stunts in cases where the conditions of a scene require their use.
This new method is unique compared to other approaches in several respects, including the fact that any face used in the set can be swapped with any recorded performance, which makes it relatively easy to re-imagine the actors at the request. The other is that it turns on the conditions of contrast and light in a stage of composition to guarantee that the actor seems to be really present in the same conditions as the scene.
You can check the results for yourself in the video below (as the researchers point out, the effect is actually much better in moving videos than in still images). There is always a hint of the “mysterious valley” effect going on here, but the researchers also recognize that, calling it “a major step towards the exchange of photo-realistic faces that can successfully bridge the mysterious valley” in their article. Basically, it’s a lot less nightmare fuel than the other attempts I’ve seen, especially when you’ve seen the side-by-side comparisons with other techniques in the sample video. And, most notably, it operates at a much higher resolution, which is essential for actual use in the entertainment industry.
The examples presented are a very small sample, so it remains to be seen to what extent this can be applied. The subjects used seem to be mainly white, for example. Also, there is always the question of the ethical implication of any use of face swapping technology, especially in video, as it could be used to make credible videos or photographic “ evidence ”. of something that didn’t actually happen.
Given, however, that technology is now being developed from multiple neighborhoods, the time for debate on the ethics of its development and exploration has largely passed. Instead, it is welcome for organizations like Disney Research to follow the academic path and share the results of their work, so that others concerned about its potential malicious use can determine ways to report, identify and protect against bad actors.