Continuously varying frame rates – increased video quality and new opportunities for video artists

Computer scientists from Saarbrücken, Germany, developed a technology that enables to emulate the perception of continuously varying frame rates in local regions of a video, hence giving movie production and video artists a new design tool for motion pictures. Using this pioneering video processing method, one can freely adjust sections within a video, giving the opportunity to balance in a video between localized high brilliance and richness of details on one hand and global aesthetics, e.g. the „cinematic-look“, on the other hand. Thanks to a skillful implementation there are no unpleasant visual artifacts between the local subfields, such as flicker or judder.

Often the limited frame rate is the cause for visual artifacts in a motion picture or video. The frame rate determines how far an object with a given velocity moves between two consecutive frames. Fast motions in the picture can cause deluding effects such as blur, flicker, or judder when the frame rate is low, and also other optical illusions such as the so-called wagon-wheel effect (i.e. a spoked wheel that seems to spin backwards compared to the actual direction of rotation) can be related to a frame rate that is too low and not practical for the scene. Higher frame rates can reduce or avoid these artifacts, giving finer details or more brilliance. A higher picture quality can show moving objects more precisely, but can also give a plastic „soap opera look“ at the same time that some audience does not like.

There are examples of famous motion pictures with higher frame rates. The movie „The Hobbit“ was filmed with 48 frames per second, fps, and Disney studios has experimented with discrete frequencies as low as 6 fps within a 48 fps movie. By using varying frame rates instead of the limited set of fixed frame rates that is currently available, it is possible to emphasize or attenuate parts of the video, increase its objective appearance or artistically modify it, hereby giving motion picture artists a new means of design.

The groups of Dr. Piotr Didyk, Saarland University, and Professor Karol Myszkowski, Max Planck Institute for Informatics, both located in Saarbrücken, had the idea to emulate different frame rates by computer algorithms. „We wanted to display arbitrary regions of a video with freely selectable frame rates. One of many challenges was to not introduce any artifacts in the zones where the frame rates change, but instead ensure a smooth transformation from a subfield with different frame rate to surrounding“, says Piotr Didyk, one of the involved computer scientists and now Group Leader at the Cluster of Excellence Multimodal Computing and Communication. With this technology, it is possible to display different areas of a video at individual frame rates and the intermediate at continuously varying frame rates without any visual artifacts distracting the eye. (See http://resources.mpi-inf.mpg.de/FrameRateEmulation).

Dr. Krzysztof Templin, primary investigator in this project, former PhD student under Professor Myszkowski and Professor Seidel, and the first author of the associated publication explains: „The picture sequence is synchronized such that it results in a video stream that has the fixed frame rate of the final display device, but at the same time the appearance created by our technique corresponds to a situation in which the framerate of the display was locally changed“. Potential users of this technology are motion picture studios and video artists, also film schools showed big interest.

Currently, the scientists look for ways of exploiting this technology in real movie productions and TV sets, an international patent application was filed. This technology of emulated varying frame rates was presented to the public this summer in Anaheim, California, where from 24th to 28th July, the worldwide largest and most influential conference on Computer Graphics, SIGGRAPH, took place. A publication that is accepted at this conference (after a rigorous selection process) is a milestone for every computer graphics scientist.

Currently, the laboratory model is being developed into a commercial product. The Universität des Saarlandes Wissens- und Technologietransfer GmbH is the local partner in this, in collaboration with the Max Planck Innovation GmbH, the technology transfer unit of the Max Planck Society.

further information:

research project Varying Frame Rates
resources.mpi-inf.mpg.de/FrameRateEmulation/

Wissens- und Technologietransfer GmbH
www.kwt-uni-saarland.de/de/meta/kwt-wut.html

Max Planck Innovation
www.max-planck-innovation.de/

Background:
A film in a movie theater, a video on a computer monitor, or a news show on TV, these formats are called moving images and considered as such by the spectators. Nevertheless it is a sequence of single frames merged into a continuous motion by the human eye and brain. As long as the shift between consecutive frames is small and the frame rate high enough, the visual perception comes to its limits, hence the brain sees undisrupted movements.
For decades, movies for the cinema have been filmed at 24 fps. Since the introduction of the sound film this frame rate has become the de facto standard for motion pictures. The viewing habits of the cinema audience have been shaped by this technique; it defers from the visual impression given by television or computer monitors. Until recently, display techniques in the movie theaters (24 fps) and TV (50 / 60 half-frames per second with PAL / NTSC) are based on fixed frame rates. It is only with CRT computer monitors 1990 and later that the displays have more flexibility. Currently, LED monitors have a preferred frequency.

Contact for technological questions:

Christof Schäfer
WuT GmbH
+49.681.302-6383
christof.schaefer@uni-saarland.de

Copyright attribution:
The original movie sequence the shown figure is based on has been distributed under Creative Commons Attribution 3.0 license, (CC) Blender Foundation | mango.blender.org. It has been modified according to the technology as described in the scientific publication.