Subjective video quality

Subjective video quality is video quality as experienced by humans. It is concerned with how video is perceived by a viewer (also called "observer" or "subject") and designates their opinion on a particular video sequence and therefore related to the field of Quality of Experience. The measurement of subjective video quality is necessary since objective quality assessment algorithms such as PSNR have been shown to correlate badly with ratings. Subjective ratings may also be used as ground truth to develop new algorithms.

Subjective video quality tests are psychophysical experiments in which a number of viewers rate a given set of stimuli. These tests are quite expensive in terms of time (preparation and running) and human resources and must therefore be carefully designed.

In subjective video quality tests, typically, SRCs ("Sources", i.e. original video sequences) are treated with various conditions (HRCs for "Hypothetical Reference Circuits") to generate PVSs ("Processed Video Sequences").^[1]

Measurement

The main idea of measuring subjective video quality is similar to the Mean Opinion Score (MOS) evaluation for audio. To evaluate the subjective video quality of a video processing system, the following steps are typically taken:

• Choose original, unimpaired video sequences for testing
• Choose settings of the system that should be evaluated
• Apply settings to the SRC, which results in the test sequences
• Choose a test method, describing how sequences are presented to viewers and how their opinion is collected
• Invite a panel of viewers
• Carry out testing in a specific environment (e.g. a laboratory context) and present each PVS in a certain order to every viewer
• Calculate rating results for individual PVSs, SRCs and HRCs, e.g. the MOS

Many parameters of the viewing conditions may influence the results, such as room illumination, display type, brightness, contrast, resolution, viewing distance, and the age and educational level of viewers. It is therefore advised to report this information along with the obtained ratings.

Source selection

Typically, a system should be tested with a representative number of different contents and content characteristics. For example, one may select excerpts from contents of different genres, such as action movies, news shows, and cartoons. The length of the source video depends on the purpose of the test, but typically, sequences of no less than 10 seconds are used.

The amount of motion and spatial detail should also cover a broad range. This ensures that the test contains sequences which are of different complexity.

Sources should be of pristine quality. There should be no visible coding artifacts or other properties that would lower the quality of the original sequence.

Settings

The design of the HRCs depends on the system under study. Typically, multiple independent variables are introduced at this stage, and they are varied with a number of levels. For example, to test the quality of a video codec, independent variables may be the video encoding software, a target bitrate, and the target resolution of the processed sequence.

It is advised to select settings that result in ratings which cover the full quality range. In other words, assuming an Absolute Category Rating scale, the test should show sequences that viewers would rate from bad to excellent.

Viewers

Viewers are also called "observers" or "subjects". In order to obtain representative ratings, a certain number of viewers should be invited. This number is not strictly defined. According to ITU-T, any number between 4 and 40 is possible, where 4 is the absolute minimum for statistical reasons, and inviting more than 40 subjects has no added value.^[2] It is claimed that at minimum 10 subjects are needed to obtain meaningful averaged ratings.^[3] However, the ITU-T recommendations were designed for measuring typical picture quality encountered by a home television user, where the range and diversity of distortions tend to be quite small. Given the large ranges and diversity of impairments that occur on videos captured on mobile devices and/or transmitted over wireless networks, much larger numbers of human subjects may be needed.

Viewers should be non-experts in the sense of not being professionals in the field of video coding or related domains. This requirement is introduced to avoid potential subject bias.

Typically, viewers are screened for normal vision or corrected-to-normal vision.

Test environment

Subjective quality tests can be done in any environment. However, due to possible influence factors from heterogenous contexts, it is typically advised to perform tests in a neutral environment, such as a dedicated laboratory room. Such a room may be sound-proofed, with walls painted in neutral grey, and using properly calibrated light sources. Several recommendations specify these conditions.^[4][5]

Crowdsourcing has recently been used for subjective video quality evaluation, and more generally, in the context of Quality of Experience.^[6] Here, viewers give ratings using their own computer, at home, rather than taking part in a subjective quality test in laboratory rooms. Recently, the world's largest crowdsourced picture quality database was made publicly available by the Laboratory for Image and Video Engineering (LIVE). The new LIVE In the Wild Image Quality Challenge Database contains more than 350,000 human quality judgments from more than 8,000 human subjects. The human judgments were made on more than 1,100 mobile camera photographs afflicted by varying degrees of authentic distortions, instead of synthetically-introduced distortions as has been done in nearly every prior database.

Analysis of results

Opinions of viewers are typically averaged into the Mean Opinion Score (MOS). To this aim, the labels of categorical scales may be translated into numbers. For example, the responses "bad" to "excellent" can be mapped to the values 1 to 5, and then averaged. MOS values should always be reported with their statistical confidence intervals so that the general agreement between observers can be evaluated.

Often, additional measures are taken before evaluating the results. Subject screening is a process in which viewers whose ratings are considered invalid or unreliable are rejected from further analysis. Invalid ratings are hard to detect, as subjects may have rated without looking at a video, or cheat during the test. The overall reliability of a subject can be determined by various procedures, some of which are outlined in ITU-R and ITU-T recommendations.^[2][5] For example, the correlation between a person's individual scores and the overall MOS, evaluated for all sequences, is a good indicator of their reliability in comparison with the remaining test participants.

Standardized testing methods

There are many ways to select proper sequences, system settings, and test methodologies. A few of them have been standardized. They are thoroughly described in several ITU-R and ITU-T recommendations, among those ITU-R BT.500^[5] and ITU-T P.910.^[2] While there is an overlap in certain aspects, the BT.500 recommendation has its roots in broadcasting, whereas P.910 focuses on multimedia content.

A standardized testing method usually describes the following aspects:

• how long an experiment session lasts
• where the experiment takes place
• how many times and in which order each PVS should be viewed
• whether ratings are taken once per stimulus (e.g. after presentation) or continuously
• whether ratings are absolute, i.e. referring to one stimulus only, or relative (comparing two or more stimuli)
• which scale ratings are taken on

Another recommendation, ITU-T P.913,^[4] gives researchers more freedom to conduct subjective quality tests in environments different from a typical testing laboratory, while still requiring them to report all details necessary to make such tests reproducible.

Examples

Below, some examples of standardized testing procedures are explained.

Single-Stimulus

• ACR (Absolute Category Rating):^[2] each sequence is rated individually on the ACR scale. The labels on the scale are "bad", "poor", "fair", "good", and "excellent", and they are translated to the values 1, 2, 3, 4 and 5 when calculating the MOS.
• ACR-HR (Absolute Category Rating with Hidden Reference): a variation of ACR, in which an original unimpaired source sequence is shown in addition to the impaired sequences, without informing the subjects of its presence (hence, "hidden"). The ratings are calculated as differential scores between the reference and the impaired versions. The differential score is defined as the score of the PVS minus the score given to the hidden reference, plus the number of points on the scale. For example, if a PVS is rated as “poor", and its corresponding hidden reference as “good", then the rating is ${\textstyle 2-4+5=3}$. When these ratings are averaged, the result is not a MOS, but a differential MOS ("DMOS").
• SSCQE (Single Stimulus Continuous Quality Rating):^[5] a longer sequence is rated continuously over time using a slider device (a variation of a fader), on which subjects rate the current quality. Samples are taken in regular intervals, resulting in a quality curve over time rather than a single quality rating.

Double-Stimulus or Multiple Stimulus

• DSCQS (Double Stimulus Continuous Quality Scale):^[5] the viewer sees an unimpaired reference and the impaired sequence in a random order. They are allowed to re-view the sequences, and then rate the quality for both on a continuous scale labeled with the ACR categories.
• DSIS (Double Stimulus Impairment Scale)^[5] and DCR (Degradation Category Rating):^[2] both refer to the same method. The viewer sees an unimpaired reference video, then the same video impaired, and after that they are asked to vote on the second video using a so-called impairment scale (from "impairments are imperceptible" to "impairments are very annoying").
• PC (Pair Comparison):^[2] instead of comparing an unimpaired and impaired sequence, different impairment types (HRCs) are compared. All possible combinations of HRCs should be evaluated.

Choice of methodology

Which method to choose largely depends on the purpose of the test and possible constraints in time and other resources. Some methods may have fewer context effects (i.e. where the order of stimuli influences the results), which are unwanted test biases.^[7] In ITU-T P.910, it is noted that methods such as DCR should be used for testing the fidelity of transmission, especially in high quality systems. ACR and ACR-HR are better suited for qualification tests and – due to giving absolute results – comparison of systems. The PC method has a high discriminatory power, but it requires longer test sessions.

Databases

The results of subjective quality tests, including the used stimuli, are called databases. A number of subjective picture and video quality databases based on such studies have been made publicly available by research institutes. These databases – some of which have become de facto standards – are used globally by television, cinematic, and video engineers around the world to design and test objective quality models, since the developed models can be trained against the obtained subjective data.

Examples of these databases include the popular collection of public-domain subjective picture quality databases available at the Laboratory for Image & Video Engineering (LIVE) website. The LIVE databases were the first high quality, large subjective picture quality databases to be made available to the public free of charge and without withholding of any data portion. Recent work includes the databases by the Institut de Recherche en Communications et Cybernétique de Nantes (IRCCyN). A list of European databases related to the field of Quality of Experience can be found in the QUALINET databases.

External links

• Video Quality Experts Group
• Institut de Recherche en Communications et Cybernétique de Nantes (IRCCyN).
• Laboratory for Image & Video Engineering (LIVE)

References

1. ITU-T Tutorial: Objective perceptual assessment of video quality: Full reference television, 2004.
2. ITU-T Rec. P.910 : Subjective video quality assessment methods for multimedia applications, 2008.
3. Winkler, Stefan. "On the properties of subjective ratings in video quality experiments". Proc. Quality of Multimedia Experience, 2009.
4. ITU-T P.913: Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment, 2014.
5. ITU-R BT.500: Methodology for the subjective assessment of the quality of television pictures, 2012.
6. Hossfeld, Tobias (2014-01-15). "Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing". IEEE Transactions on Multimedia.
7. Pinson, Margaret and Wolf, Stephen. "Comparing Subjective Video Quality Testing Methodologies". SPIE Video Communications and Image Processing Conference, Lugano, Switzerland, July 2003.