The Effects of Audio-Visual Skew and Background Sounds on Human Perception of Video Content

Overview

Objectives/Goals "User Content Generation"(e.g. YouTube, MySpace) is attracting huge interest world-wide and growing rapidly as an industry. Since user generated content (UGC) undergo significant processing (transcoding, transrating, etc) before being published, two important challenges face the UGC industry:(1) Media processing introduces audio-visual skews;(2) When users record their own videos, background sounds can be introduced (similar to the cocktail party effect). This project and experiments measured the effects of audio-visual skew and background sounds on human perception. The goal was to understand the limit of audio/video skew on human perception and the combination effects of audio/video skew and background sounds on human perception. Methods/Materials Three amounts of delays were tested (0 sec, 0.5 sec and 1 sec). Also, background sounds (BG) were added to simulate real-life recording conditions. The human subjects watched a video clip, then completed a human perception survey. To create the (6) clips, I used ULead Video Studio 10. The clips with BG were produced using exactly the same clips, with a song fused in the audio channel. Materials: Original clip of an Emma Watson interview, original music clip of "Let's get loud", ULead Video Studio 10 editor, Derived video clips described above, Laptop, Headphones, and the Test Survey. Results The experiments show that audio-visual skews greater than 0.5 seconds or more significantly degrade human perception. Also, BG degrades human perception further. The importance of lip synchronization was significantly amplified when BG were added. With degraded lip synchronization, the lag of mouth shape information not only fails to help resolve ambiguity introduced by BG, but also distracts humans from attending to audio while they are trying to resolve past ambiguities (current visual with past audio). Conclusions/Discussion As video communication technologies advance, this research provides valuable insight as to the effect of audio-visual delay and BG on human perception, a starting point for further technology development on how lip-synch can be maintained well below 500ms. Perhaps, a better expression of time-stamp of audio and video in the streams and better specification on the actual transformation and mapping of time-stamps during the media processing stage could be a better approach to maximize audio and video alignment.

Awards (1)

• Fourth Place

Competition history

• CSEF 2007 Behavioral & Social Sciences · Entry S0310

Resources

• CSEF project summary ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Behavioural and Social Sciences projects
• CSEF Behavioural and Social Sciences projects
• Every CSEF 2007 project
• CSEF Fourth Place winners