Algorithmic composition and machine learning in film editing

Project start:
2017
Project completion:
2018

Film editing and storytelling are tasks that are difficult to automate because of their complexity and their demand on creativity. They require a deep understanding of human emotions and dynamics, of internal attitudes and effects based on dramaturgy. Nevertheless, there is a wide field of application for algorithmic composition and machine learning in film editing: It ranges from assisting in the editing of classical films to experimental artistic work and interactive computer games.

Jonathan Frank focuses on the artistic applications of algorithmic composition in the field of film editing. To this end, he uses machine learning technologies and develops an automatic editing machine in a practical experiment. Based on learned material, this machine assembles new material into a film. He puts his project into practice using the programming language "Pure Data", which Jonathan Frank learned specifically for this project. He writes programs that use Markov chains - special stochastic processes - to generate new music from existing pieces of music. Algorithmic music generation serves him as a testing ground for machine learning in film editing, since relations between sounds are easier to formalize than relations between images.

In his bachelor thesis, he examined improvisation in film editing based on diagrammatic scores. These scores can generally be seen as a precursor to computer-generated production of artistic creation. The topic of "improvisation in film montage" was a first attempt for him to answer the question of how to behave in editing when there is no given set of rules. With the approach of algorithmic composition, he now looks at this question from an opposite perspective.

Project lead: Jonathan Frank