Organisation internationale de normalisation

5.2.3.2Revision of CE methodology document

The Chair brought up a revised version of N7140, Revised core experiment methodology for MPEG audio. The following captures the discussion of the open issues in that revised document.

The CE methodology document should document the following means of statistical analysis and indicate the appropriate context for their use.

• 
  Two-sided 95% confidence interval on the mean based on the assumption of Gaussian PDF. Appropriate for large sample sized and multiple systems under test.
  
• 
  Two-sided 95% confidence interval on the mean based on a small sample set (i.e. t-test analysis). Appropriate for small sample sized and multiple systems under test.
  
• 
  Single-sided 95% confidence interval on the difference between means based on the assumption of Gaussian PDF. Appropriate for large sample sized and two systems under test
  
• 
  Single-sided 95% confidence interval on the difference between means based a small sample set (i.e. t-test analysis). Appropriate for small sample sized and two systems under test
  

The CE methodology document should document the following means of subjective quality assessment and indicate the appropriate context for their use.

• 
  BS-1116 triple-stimulus hidden reference. Appropriate for assessment for near-transparent systems.
  
• 
  MUSHRA. Appropriate for assessment for intermediate quality systems
  
• 
  A/B Comparative test. Appropriate for two systems under test where maximum sensitivity is needed but there is no need for an assessment of the absolute level of subjective quality.
  

The subjective quality assessment method and the statistical analysis method should be selected by the consensus of the Audio Subgroup as appropriate for each work item but may be changed for a particular Core Experiment.

The subjective performance data from test sites should be examined for consistency. The data from a majority of test sites should show an increase in performance for the CE technology.
Test items – The 12 CfP test items should be used in the CE process. If the CE technology or circumstances warrant, different test items can be used, but this should be decided on a CE by CE basis.
Operating points – The CE proponent must show improvement at least one of the nine operating points and no degradation at the remaining operating points.