Scholars of film and television are working with increasingly vast digital archives. While artificial intelligence can help sort through these collections, most existing tools fall short of what humanities researchers need.

Current AI models can identify objects or faces on screen, but they can’t identify how or why a scene was filmed in a particular way or how editing choices shape a story.

University of Richmond professors Lauren Tilton and Taylor Arnold are answering that question.

In partnership with colleagues at UC Berkeley and Bowdoin College, Arnold, professor of data science and statistics, and Tilton, professor of digital humanities and director of UR’s Center for Liberal Arts and AI, were awarded a grant from Schmidt Sciences to develop AI models that analyze film and television. 

Over the next two years, this team of experts in data science, computer science, and film studies will develop new AI models to analyze camera movement, narrative storylines, and how editing and dialogue are used to create meaning. The Richmond team will track the evolution of patterns across long-running television series, tracing how storytelling techniques evolve.

“The project represents significant advancements in both the humanities and AI,” said Tilton. “Our research will further the ability of AI not only to recognize objects in moving images but to analyze patterns, an important step forward for all scholarly communities working with digital media.”

Current AI models might be able to show you all the scenes containing a particular object in a film, explained the scholars. Their challenge is to develop new generative AI tools that can identify techniques in films, such as camera movements.

The patterns could help scholars explore questions such as how close-ups influence psychological engagement, how pacing affects narrative tension, and how visual and audio techniques vary across genres.

The project team plans to share its findings by hosting a symposium for film and television specialists and related scholars, publishing articles, and releasing open-source software for others to use with their own collections.

Arnold and Tilton are no strangers to developing computational methods that further the field of digital humanities. Their work also includes UR’s Distant Viewing Lab, which uses computational techniques to analyze visual culture on a large scale. A toolkit offered through the Lab gives users access to AI-powered computer vision models with no coding or special hardware required, improving accessibility to these digital humanities tools.

“Generative AI has the potential to strengthen humanities study,” said Tilton. “Traditional humanities are focused on critical thinking, engaging with art and literature, developing imaginative possibilities, and learning to think historically – AI tools like the ones we’re developing can help scholars more deeply engage with all these skills.”