Machine learning now helps drones to position themselves on film sets. Here’s how. +
It takes years of work to become director of photography. Unless, obviously, you are a drone.
Researchers at Carnegie Mellon University have just developed an aerial cinematography system that takes into account human visual preferences to allow drones to make cinematographic and artistic choices while filming scenes independently. The system does not require GPS tags to locate targets or previous maps of an environment.
Drones are a boon for filmmakers. They significantly reduce the costs of aerial photography, which previously required the chartering of helicopters or aircraft. But this ease of access also has a disadvantage. Many filmmakers use drones in their work, and often the shots seem to be shot, especially if they are shot with a stand-alone drone.
“We put the power of a director inside the drone”
“We put the power of a director inside the drone,” says Rogerio Bonatti, PhD student at the CMU Robotics Institute. “The drone is positioned to record the most important aspects of a scene, it understands the context of the scene – where the obstacles are, where the actors are – and it actively reflects on the points of view that go to make the scene more visually interesting, and it also works on flying safely and not crashing. “
As a spokesperson for CMU points out, the notion of “artistically interesting” is quite moving. Instead of trying to obtain artistically satisfying plans, the system was trained using a technique called
In one case study, subjects watched scenes on a photorealistic simulator that changed between front, back, left and right perspectives. Users rated the scenes based on their visual appeal and artistic interest. The system learned these preferences.
Generalize specific behaviors
For example, the system has learned that a backshot, by far the dominant scene captured with drones, becomes annoying to viewers after a while. Filmmakers often change angles to keep the shot interesting. However, a change of view too frequent quickly causes fatigue in the viewer. The trick is to do it right.
The CMU team has generalized specific behaviors, allowing the drone to apply lessons learned in a scenario to different shooting scenarios. For example, lessons learned for the plans of an actor walking in a narrow corridor could be applied to similar ambulatory scenes, such as someone walking on a forest trail.
“Future work could explore many different parameters or create custom artistic preferences based on a director’s style or genre,” says Sebastian Scherer, an associate professor of research at the Robotics Institute.
The system could be used outside the cinema
The aerial system also helps maintain an unobstructed view of the actor, avoiding so-called occlusions. “We were the first to find new ways to treat occlusion that are not only binary, but can actually quantify the severity of the occlusion,” said Bonatti.
According to researchers at CMU, the system could be used outside the cinema, including in the entertainment and live sports broadcast. It is also possible that police services that currently use hand-held drones to monitor crowds and understand traffic patterns can benefit from this research.
“The goal of the research is not to replace humans, we will always have a market for highly qualified professional experts,” Bonatti said. “The goal is to democratize drone cinematography and allow people to focus on what really matters to them.”