Publications

The proliferation of AI in movie production is still in its early days. The industry is currently dominated by technologies that revolve around computer-generated imagery(CGI) and visual effects. 

The integration of AI promises some quick positive effects in some phases of production. However, some fundamental questions arise from the use of AI. Here are some benefits and drawbacks of using AI in movie making.

Movie makers can derive compelling film storylines using AI-driven algorithms. It’s possible to analyze large datasets to identify patterns and trends across different points in the production. For instance, insights generated can help create relatable characters and grand narratives for scriptwriters.

Filmmakers can utilize AI to generate complex visuals to bring to the audience a level of realism. Additionally, it's now possible to adjust the mouth movement of actors while translating the movie to another language to make it more natural. This makes it easier to translate and distribute movies in different languages. 

Other ways that AI enhances the quality of movies produced include:

• De-aging actors - 80-year-old Harrison Ford was de-aged for the movie “Indiana Jones and theDial of Destiny” with stunning results [5].

• Machine learning(ML) algorithms to enhance visual and audio.

With AI-powered tools, filmmakers face less tedious manual processes such as retakes and editing, which can be time-consuming. For instance, using computer vision and facial recognition, production workers retouch shots efficiently. 

Companies like Colourlab [6] offer AI-driven software that automates mundane editing tasks leaving content creators to only add the finishing touches. Such tools provide the capability to easily balance film color, use smart filters, and edit images.

ML can track and analyze the movement of actors and objects on virtual sets. This helps the production team generate complex special effects and natural-looking scenes that would be challenging using traditional methods. Virtual production with AI offers a safer way to produce movie scenes. It eliminates the need for actors and crew members to make dangerous maneuvers or perform in risky environments.

There are many ways in which the use of AI in film production leads to impacting the cost. Savings can be derived primarily across different phases, including virtual production, predictive analytics, location scouting, and post-production efficiency.

For instance, virtual production helps crew members produce visual effects in a room LED stage in real time. 

Companies like Cuebrick help filmmakers generate scenes quickly with Stable Diffusion as its image generation engine [7]. This makes it a quick and cost-effective method to build sets in real-time, work which would take an entire CGI team several weeks. Previously, action sequences had to be integrated with CGI after filming.

AI relies heavily on algorithms and data-driven decisions, which can lead to homogenous ways of doing things. This is especially true for storytelling and scriptwriting, with AI unable to conjure original ideas. Over-reliance on AI could stifle creativity and originality, leading to one-dimensional content while diluting unique perspectives.

As different processes in film production are automated or changed by AI integration, there’s a potential risk of human displacement. Its impact on traditional roles, such as script analysts and color correction, can reduce the demand for professional input. 

Voice cloning, deep fakes, and synthetic video and image production are some areas actors and filmmakers will have to collaborate with AI and may require reduced human input. For instance, the use of AI-powered editing tools may reduce the time human editors work on a project.

With the development of new standards and tools with AI capabilities, there’s bound to be a shift in skill requirements. Film-makers will need to learn how to collaborate with different AI-powered tools and leverage them.

AI models are trained on vast amounts of data, including texts, images, audio and videos. As these models are trained using existing data, some scrapped off the web,  AI systems may produce similar content inadvertently. This may lead to issues about intellectual property through unintentional duplication. 

Intellectual property infringement is also a pressing concern. For instance, Getty Images and a group of artists have instituted a lawsuit against Stability AI, a generative AI company that offers image generation. The plaintiff accuses Stability AI of using its copyrighted images without consent and compensation [8].

The proliferation of AI in filmmaking makes it possible to manipulate audio, video, and images aiming to produce content that appears genuine. It can lead to the creation of scenes and narratives that never happened. It’s, therefore, possible to manipulate what actors say and do on a movie set, leading to misrepresentation.

The filmmaking industry has used AI-driven solutions since the 2000s, especially in CGI and special effects. Technological advancements such as 3D graphics enabled filmmakers to make compelling special effects. Progress in ML and computing power has been essential in areas such as facial recognition and motion tracking commonly used in movie sets.

The use of AI found a place in special effects and CGI early on, even in popular movies such as Tron(1992). 

Tron is considered a pioneering movie for using CGI to create special effects. While not necessarily using AI as we think of today, the production used advanced CGI to create visually appealing environments. The movie inspired other filmmakers to embrace technology in production, exploring different ways technology could help.

Furthermore, the film "The Irishman" used AI to de-age Robert De Niro, Joe Pesci, and Al Pacino. Again, this helped filmmakers create de-aging effects without manually tracking the actors' faces [9].

Advancements in ML and deep learning algorithms have enabled the use of AI techniques for different processes in film production. Some of the AI techniques the film production pipeline uses include.

Natural Language Processing(NLP): AI systems can now understand human language, generate text, and much more. The impact of NLP is felt in script analysis, subtitle generation, dialogue writing, and storyboarding.

Generative Adversarial Networks(GANs): This generative modeling approach uses deep learning techniques such as convolution neural networks. This has several use cases, especially in image synthesis. Filmmakers can generate realistic outputs depending on the input data. For instance, it can be used in DeOldify models, which apply color to recognized images and patterns [10]. 

Generative modeling techniques: Different softwares use this architecture for video and voice generation. Such techniques include variational encoders, recurrent neural networks, and conditional generative models. For instance, Respeecher, a voice cloning service, uses deep generative modeling techniques for audio synthesis [11].

Computer vision: this technique has been infused across different points in the movie production pipeline. It can be used in virtual production, motion tracking, face tracking, and scene analysis.

Movie scripts, and storylines are every movie's backbone, enabling filmmakers to weave compelling narratives and scenes. AI-powered software can process copious amounts of data from past films, scripts, and books. This allows it to output different unique plots, dialogues, locations, props, and even characters based on user input. 

In 2016, an AI named Benjamin scripted a short movie called Sunspring [12]. The AI used Long short-term memory recurrent neural network. It learned from content written between the 1980s -1990s, and could output scripts depending on the preset requirements. 

From the generated story, it’s also possible to automatically extract elements such as dialogue, main characters, location, and the different prompts needed. Using NLP methods, filmmakers can also determine the audio and visual effects coupled with the required scenes and number of shots. 

Companies like Botnik and Taskade are currently offering movie script generation services. They rely on NLP and ML to generate different script elements [13]. Scriptwriters can use prompts to generate ideas, outlines, and full-length scripts for short films. 

Other Startups, like Belgium-based ScriptBook, have carved a niche in movie script analysis, AI-driven content validation, and story generation [14]. Their software uses predictive analytics, deep learning, ML, and NLP to predict the commercial success of a film based on a script.

The pre-production phase involves various activities, including scheduling, financing, and location scouting. The use of AI helps film producers save resources and make data driven decisions. The ability of AI-powered programs to analyze large datasets and identify patterns make it possible to extract information and output recommendations. 

For instance, filmmakers can get insights about the target audience, including finer details like gender and age. By using ML to analyze datasets, producers can derive the profile of their potential audience with high accuracy.

World-renowned movie studios like 20th Century Fox have used ML to predict audiences. The team created Merlin Video, a computer vision solution that learns the dense representation of their movie trailers. The tools can then help predict a movie’s future audience based on its trailer [15]. 

Other activities, such as storyboarding, also benefit from AI integration. AI-driven storyboarding tools allow text input, such as scripts describing different scenes and shots. The algorithm uses NLP to understand the user input and computer visuals to output the corresponding visual representation. 

It’s possible to use these tools to offload the initial design brainstorming. The tools will often produce inclusive designs while limiting outputting of edge cases. Examples of AI-powered storyboarding software include Krock and Storyboardhero [16].

AI can also help determine the best actors who fit certain narratives based on metrics from previous performances. Additionally, it can be used effectively for scheduling, considering the multiple variables like actor availability, logistics, and location readiness that need to align simultaneously.

Tools like RivetAI by End Cue have a scheduling option that considerably impacts choosing the optimal path for movie production [17]. RivetAI, for instance, deals with the combinatorial optimization issue, using previously solved scheduling problems and other AI techniques, such as mixed integer linear problem. The company states that it expects film projects to save in double-digit percentages in time and cost efficiencies compared to when scheduling optimization was done by hand. 

Mitchel Blocks, an Academy Award-winning executive producer, thinks AI is the easy way to simplify the pre-production process, from budgeting to casting [18]. 

Companies like Cinelytic provide solutions that influence the pre-production and movie planning phase. It primarily offers analytics and a predictive intelligence engine to help movie studios greenlight movies and better determine release dates [19].

AI-driven tools are also influencing the film production phase making the process more efficient. For instance, crew members can use AI tools to improve camera angles and lighting, create realistic visual effects and select the best takes from a scene. 

ML and  AI play a crucial role in motion control camera systems. This technology allows crew members to produce precise, repeatable action sequences with different production elements. It's possible to detect characters, analyze scenes and optimize camera movements to achieve a production goal using AI-powered motion systems. 

Companies like Axibo develop camera motion systems using deep learning and AI for object detection and voice recognition. Its inbuilt AI processing unit helps the system learn, adapt and listen, supporting it with auto-shooting capabilities [20].

The use of AI is also prominent in film post-production. It can be used in film editing, making trailers, and special effects. Here are some areas where AI-powered tools are revolutionizing movie post-production.

Through advanced ML techniques and neural networks, AI is now influencing various stages of the VFX pipeline. New tools are allowing production crews to push creative boundaries, enabling them to generate complex simulations ranging from special weather effects to explosions. Combining VFX and CGI helps filmmakers make production have better diversity and realism while reducing production costs and time associated with VFX. 

For instance, the movie Avengers Infinity Wars combines AI and CGI. Kelly Port, the visual effects supervisor responsible for producing the main character Thanos, admitted to using ML techniques. The team fed still scans, motion scans, and helmet-cam data to an ML algorithm that produced the fitting and high-resolution results [21].

The movie”The Irishman” used different deep learning neural networks to de-aging De Niro, Joe Pesci, and Al Pacino. The AI algorithm mapped a younger version of the actors faces, creating their younger version. The results were visually stunning and realistic.

Film trailers present filmmakers with a way to build movie anticipation, giving the audience a preview of what to expect. Movie studios, therefore, go a long way to create intriguing trailers. 20th Century Fox partnered with IBM Research to develop a trailer for a movie called “Morgan”. Using ML techniques and IBM Watson API, the team used the tool for visual, audio, and scene composition analysis [22].

Companies like Topazlabs have AI models that improve both video and still pictures. They use deep learning techniques to denoise images while recovering essential details. For video production, the tool supports video enhancements through upscaling, deinterlacing, and motion interpolation [23]. Their AI- solution can also enhance lower-quality footage by removing macro blocking, moire, and other issues that affect such footage.

There’s also a revolution in rotoscoping through AI, tackling the problematic area of extracting individual elements from background environments. The process has been labor-intensive, requiring animators to trace over objects in frames. With the infusion of AI, artists can automate the process while improving accuracy.

The AI-driven solution will typically use computer vision and image segmentation techniques. An image segmentation algorithm extracts distinct multiple entities from a background [24].

A company known as Weta Digital has already made use of AI rotoscoping in the making of a movie known as “The Mandalorian” [25].  Its use resulted in savings in time and cost while enhancing the overall visual quality.

The use of AI techniques such as deep learning can be used to analyze facial expressions in real time. This information can then animate computer-generated characters, creating realistic facial expressions. This process produces high-quality digital characters while making the process more efficient and less expensive.

Additionally, AI can be used for blendshaping, a technique popular for making transitions in 3D animation.

AI-powered video editing tools are making it easier and more efficient for editors. Analyzing the shots, detecting patterns, identifying the best photos, and enhancing the video is now possible using AI-powered solutions. Some of the editing activities include color grading, subtitling, and upscaling.

Movies where AI was used for editing include

1. The Irishman - AI techniques were used to de-age the main actor, making them look younger

2. Tenet - AI was used to create a complex time-bending effect

3. Avengers: Infinity Wars - AI was used to create Thanos using a custom software called Masquerade [26].

4. The Lion King - The movie used deep fake AI

5. Avatar: The Way of Water - AI was used to create special effects

According to Andrew Page, the director of advanced technology and entertainment at Nvidia, AI has already shown its mettle by automating different aspects of filmmaking. He especially points out the laborious and repetitive tasks such as metadata tagging, captioning, and rotoscoping as areas where AI has had a valuable impact on editing assistance and workflow optimization [27].

When integrated with AI-capable cloud solutions like AWS, storage solutions like EditShare can optimize workflows by automating tagging shots with metadata. The tool automatically organizes clips, assisting editors in quickly finding the shots they need. Additionally, editors can manage and find clips via dialogue through the power of ML [28].

AI-powered editing tools like Adobe’s Premiere Pro enable editors to manipulate videos via text-based editing. The tool allows generating a transcript to create your rough copy by copying and pasting text. Editors can find specific moments through keyword searching, delete pauses and gaps. Creating visual continuity is quick due to the ease of sequencing clips through morph cuts [29]. 

Furthermore,  there are AI tools designed to manipulate sound and compose music. Premiere Pro offers two important features for audio manipulation: Remix helps sound engineers sync video and audio while Auto Ducking harmonizes background audio and dialogue. These capabilities enable filmmakers save hours of manually dealing with audio and video [30]. 

Adobe Firefly can be used to accelerate workflows across the production lifecycle dramatically. This is through natural language prompting to alter scenes while offering creative possibilities [31]. It’s now possible to issue a prompt like “make the scene brighter,”  which would lead to changing the scene. 

AI also plays a crucial role in translating and re-recording dialogue in different languages for foreign consumption. With ML techniques, AI-powered software like Speechify can mimic an actor’s voice and synchronize the dubbed voices with the actor’s mouth movement [32]. 

Finally, there are multiple AI-driven solutions that can help film producers create music. For instance, Sony has developed Flow Machines, a software that utilizes ML and music analysis technology for automated music composition [33].

There are AI-capable tools for specialized applications across film production processes. AI has found applications in movie restoration, marketing, and transcription.

Using AI-driven solutions, it's possible to remove flaws from old films using deep learning techniques. The process could involve noise reduction, color correction, upscaling,  and frame interpolation.

Dennis Shiryaev transformed a short film called “A Trip Down Market Street” into 4k quality. He used an algorithm called DeOldify, which understands how objects are colored. The solution then handles colorization on old movies, applying colors to different objects in black and white films [34]. 

Deep learning neural techniques can upscale old movies with low and poor resolution. One such method is training an algorithm using pairs of high and low-quality images. The algorithm learns different patterns and can transform parts of low-quality images by analyzing the pixels and adding what is missing.

Shiryaev used an algorithm known as DAIN developed by Yang, a computer scientist professor at the University of California. The old film Shiryaev was working on had a frame rate of 16 frames per second. The algorithm predicted and generated new frames amidst the old original frames. The overall effect is much smoother transitions in the film [35].

Other deep neural network techniques involve training an algorithm using downscaled HD movies. The neural network consequently learns to inject missing details into old film [36].

Speech-to-text analysis allows the correct dialogue transcription while checking which character to attribute a specific section of dialogue depending on lips movement. 

The use of facial recognition enables producers to map speech to the correct characters.

The era of greenlighting movies based on human feelings is over. Audience insight through predictive analysis can allow filmmakers to understand which elements resonate most with their focus audience. AI predictive analysis empowers film producers to fine-tune their movies by exploring patterns and trends from data using ML.

Trailers, marketing, and release data buzz are ways films generate expectations for a movie. AI can help optimize release strategies by analyzing previous data, media buzz, and trends in the market. Companies like Cinelytic provide distribution and release analytics with a way to predict movie success by cross-referencing key features of the movie against historical data [37]. 

Recommendation systems offer movie watchers personalized content and services. Netflix and Hotstar are great examples of movie streaming services that deploy recommender systems. AI algorithms, such as content-based recommender systems, can analyze a user’s preferences, like genre, viewing patterns, and social media interaction, to offer personalized recommendations.

The proliferation of AI in the film industry presents several challenges. SAG-AFTRA union president said in an interview, “We are all in jeopardy of being replaced by machines”. This stems from his fears about the abilities of AI, and he advocates for urgent control of using the tech [38].

Film actors are concerned that their data, voices, and images can be used without their consent or for illicit purposes. Data such as actors’ images, movements, and voices captured during film production could be used to produce other films. For instance, popular actors like Gadot and Scarlette Johansson have found their faces used in movies with explicit content through deepfake methods. 

SAG-AFTRA union boss Duncan Crabtree was quoted saying that one of their major issues for the 2023 strike was the possibility of production houses using images and likeness of background actors. The information would be used for eternity in other projects without consent or compensation. 

There’s also the issue of using copyrighted works to train AI models. Creators and performers argue that there must be a compensation system whenever their work is used for this purpose. In an interview, Herchenröder, the executive director of German Screenwriter Association, called for creators to receive compensation transparently for legally protected work [39].

To mitigate these risks, copyright and intellectual property laws that govern the correct usage of actors’ images and voices must be instituted and adhered to. Additionally, movie houses must compensate and get consent whenever they use actors’ characters and voices.

The overreliance on AI techniques without the human touch can lead to lackluster productions without human nuances that make movies have originality. Due to learning from previous datasets to identify patterns and trends, there’s a risk of homogenizing content and diluting spontaneous creativity.

To mitigate this challenge, AI-assisted production must strike a balance to enhance the creative process without overshadowing emotions and unique perspectives.

It’s possible to use AI and CGI to create digital clones of dead actors and integrate them into a movie set. Some notable actors whose clones have been recreated for different movie roles in Hollywood include Paul Walker, Carrie Fisher, and James Dean, who died in 1955 [40].

Ethical questions abound. For instance, who owns the image rights, voice, and character of digitally cloned actors? Additionally, what about overexploitation beyond the agreed frameworks, for example, creating sequels or misrepresenting their work and life? 

There’s also a real fear of actors and performers losing their jobs to the dead. Actor Tom Hanks reckons that his performances could go on and on even after he passes on [41]. This is especially true for voice actors who are threatened by this situation. Friedlander, the president and founder of the National Association of Voice Actors(US), voiced his concern, fearing that digitally resurrected voice actors could take over voice actors' job [42]. 

For instance, Mel Blanc, who voiced many Looney Tunes characters like Mickey Mouse, could easily take over jobs or deny employment to other actors in subsequent productions.

Developing and integrating AI solutions in cinema and moviemaking is an expensive undertaking. For instance, Netflix spent $651 million in technology expenses in 2017, with a chunk of this going to its recommendation system [43].  

Some of the costs involved in developing AI solutions include hardware, software, labor, and infrastructure. Costs will arise from integrating the solution into existing workflows and licensing costs.

There are also a myriad of technical challenges that face the development of AI software. These include data quality, quantity, and computation resources.

The ability to create scripts and stories from AI has significant benefits, including increasing productivity. However, it cannot reproduce stories from reasoning while infusing emotions. It’s, therefore, imperative that AI complements human creativity rather than replaces it.

The possibilities of AI in filmmaking are numerous and diverse. For instance, a groundbreaking AI technique, Neural Radiance Fields(NeRF), first published in 2020, is already promising advancements in virtual production [45]. It aids in the generation of detailed 3D models from 2D. The production crew can use the tool to create high-resolution and realistic 3D scenes. 

Companies like Nvidia have developed solutions around this research. The instant NeRF software uses AI and NeRF to estimate how light behaves in the real world. This enables artists to reconstruct 3D scenes from several images taken from different angles opening up new possibilities for photorealistic visual effects [45]. 

Researchers are working on using NeRF to create dynamic backgrounds and animations. This will help VFX artists produce evolving and interactive scenes. 

Another area of advancement is using AI in visual effects and StageCraft virtual production. This is being championed by Industrial Light and Magic(ILM), who developed StageCraft to help in creating virtual photorealistic digital sets. ILM is using NVIDIA’s AI-enabled DeepSearch, aiming to make it easier for filmmakers to develop ideal sets and lighting [46]. 

There is also increasing development in AI-driven live interactive cinematic experiences. Combining live action and AI to create interactive content is now possible. Viewers can interact with fictional characters, affecting the movie's plot. Arias Studio is an entertainment platform that offers viewers interactive content through immersive and virtual production pipeline technology using AI [47].

There is increasing adoption of AI by movie studios and the film industry in general. From streamlining production processes to assisting in decision-making, AI is becoming a crucial tool in movie production, helping increase productivity and efficiency.  This means that AI can contribute to lowering costs, faster production times and help smaller teams produce high-quality movies at low budgets.

There is a need for AI and movie producers to work collaboratively to augment the creativity and technical capabilities of movie producers. 

However, there are unique challenges that are facing the use of AI in movie making. Chief among them is the digital cloning of images and voices of dead actors. Questions still linger on who owns the rights to works by digitally cloned actors. 

We are in the early days as the use of AI evolves and developers continue to output AI-enabled applications to enhance the different workflows in film production. The future promises more incredible things, especially with companies working to make it easier to quickly produce videos, virtual sets, and immersive personalized viewership. AI techniques like NeRF promise to introduce novel ways to create 3D scenes from 2D images, making work easier for crew members.

AI and human interaction in the cinema and movie-making industry are bound to create an exciting future. As more filmmakers realize the benefits and possibilities of using AI in their projects, we can expect a significant shift towards a collaborative partnership in film production in years to come.

[1] AI is here, and it's making movies. Is Hollywood ready? - Los Angeles Times

[2] Why We Strike | SAG-AFTRA Strike

[3] Motion Picture Industry Market | Size, Growth | 2022 - 2027

[4] Movies And Entertainment Market Size Report, 2030

[5] 'Indiana Jones 5': How ILM's VFX Helped De-Age Harrison Ford - Variety

[6] Features - Colourlab.ai

[7] Cuebric: The Breakthrough Generative A.I. Tool For Virtual Production | by Pinar Seyhan Demirdag

[8] Getty Images Sues Stability AI Over Art Generator IP Violations

[9] The Irishman | Industrial Light & Magic

[10] DeOldify: GAN based Image Colorization | by Merzmensch | Towards Data Science

[11] Respeecher

[12] Movie written by algorithm turns out to be hilarious and intense | Ars Technica

[13] AI Script Writing Template | Taskade

[14] ScriptBook

[15] How 20th Century Fox uses ML to predict a movie audience | Google Cloud Blog

[16] Storyboard Hero

[17] RivetAI

[18] How Artificial Intelligence Helps (and Hurts) Filmmaking

[19] From Greenlight through Release, Cinelytic Provides Actionable Insights in Real-time

[20] Tilt | Slide: A.I. Powered Camera Assistant by Axibo Media — Kickstarter

[21] Avengers: Infinity War's effects behind Josh Brolin's Thanos - CNET

[22] IBM Blog

[23] DeNoise AI - Remarkable Image Noise Reduction

[24] Artificial intelligence in digital visual effects

[25] AI Tools for VFX | MOD VFX Blog

[26] AI gives Thanos a soul in 'Avengers: Infinity War'

[27] & [28] This is how AI is transforming the post-production industry

[29] & [30] AI video editing in Premiere Pro | Adobe

[31] Adobe Firefly – Generative AI for Everyone

[32] Speechify

[33] Flow Machines | Sony Design

[34] & [35] AI Magic Makes Century-Old Films Look New | WIRED

[36] AI movie restoration - Scarlett O’Hara HD - deepsense.ai

[37] Cinelytic

[38] Actors are really worried about the use of AI by movie studios – they may have a point

[39] AI and film: Actors and writers fear obsolescence – DW – 07/21/2023

[40] & [41] & [42] How AI is bringing film stars back from the dead - BBC Future

[43] How Netflix's AI Saves It $1 Billion Every Year | The Motley Fool

[44] Revolutionizing Virtual Production: How Neural Radiance Fields Will Supercharge Production Pipelines

[45] NVIDIA Instant NeRF Sweepstakes

[46] ILM Presents: Leveraging AI in Visual Effects and StageCraft Virtual Production | NVIDIA On-Demand

[47] Aria Studios, Inc.

Written by Baptiste Aelbrecht & Jacques Mojsilovic