International organisation for standardisation organisation internationale de normalisation

5.6Test Conditions & Evaluation

m23779	Yunseok Song, Cheon Lee, Yo-Sung Ho	Comments on Test Conditions of 3D Video Coding
m23908	Philippe Hanhart, Francesca de Simone, Martin Rerabek, Touradj Ebrahimi	3DV: Objective quality measurement for the 2-view case scenario

m23779 (GIST) resulted in three recommendations:

• 
  It is noted that there are non-existing intermediate views for some sequences. This is only considered to be an issue if we use the original views as a reference. Currently, we generate a synthetic reference for comparison.
  
• 
  To consider the possibility of specifying testing with QP less than 25, but no evidence was provided that suggests that the current QPs are too high. It was decided to keep the existing QP ranges.
  
• 
  That clarification was needed on the data to be provided. This was editorial, and agreed to be worked on offline.
  

m23908 (EPFL) reported that study was conducted to determine how well the subjective quality scores of the CFP evaluation correlate with objective measures. The results show that the PSNR of the synthesized view has the lowest correlation compared to the average PSNR of decoded views or the PSNR of a decoded view alone. Evaluations in terms of Pearson linear correlation and Spearman rank correlation were provided.

The reason for these findings is not completely clear; it was suggested that one possibility is that stereoscopic masking is playing a role. It is not recommended that decoded views be considered as the sole metric for evaluation or that synthesis quality should not be used. The contribution just indicates that caution is needed for these findings and requests the group to be aware of this.

Qualinet is also working on additional objective metrics to better measure and evaluate 3D quality. Qualinet also requested permission to use the CfP evaluation response for research. There were no objections raised by the group, and it was agreed that a liaison response would be drafted.

The potential shortcoming of using only PSNR of decoded views was discussed: this does not account for depth error. Use of the decoded view could be appropriate provided that a minimum level of depth quality is provided or assumed.