International organisation for standardisation

USAC CE Process

The Chair made the following remarks concerning the USAC CE process and concerning some CEs in particular.

Now that the collaborative development phase of the USAC standard is nearly at an end, as the Audio Subgroup anticipates USAC progressing to FDIS at the 97^th meeting, it is appropriate to review how well the USAC CE methodology has served us in developing this standard.

During the course of collaborative development, there was considerable discussion concerning the lack of a common encoder source code platform for use in CEs. The Call for Proposals clearly states that the RM0 proponent and subsequent successful CE proponents must submit or integrate their encoder source code such that the encoder is conformant (to the current RM) but not necessarily able to produce the exact bitstreams (i.e. level of performance) as was demonstrated in the CfP or CE. Because of this, it was envisioned to create, as a collaborative effort in the Audio Subgroup, a common encoder source code base that would be of reference quality and hence could be used in the CE process as an alternative to the proponent Reference Quality Encoder.

Now, at the end of the collaborative process, it is evident that the common encoder effort did not play a role in the CE process. Nevertheless, we envision it will be developed to have a level of performance such that it would be appropriate for inclusion in a future “follow-on” USAC Verification Test, although perhaps at a subset of operating modes.

A second aspect of the USAC CE process is that N10664, “MPEG Audio CE methodology” states in part that:

The acceptance of a core experiment is decided by the consensus of the Audio subgroup, based on the requirements described in Annex B.

Annex B states, in part:

• 
  New technology needs to demonstrate SIGNIFICANT IMPROVEMENT, where such improvement is determined by consensus of the Audio subgroup when considering all evidence including subjective audio quality.
  

Hence the CE Methodology used in the USAC collaborative phase has considerable ambiguity, in that outcomes are determined by the consensus of the Audio subgroup when considering all evidence.

During the course of the collaborative phase, CE proponents have repeatedly expressed concern that it is very difficult for them to predict the outcome of the check-phase of the CE process even after all check and cross-check performance information (e.g. listening test results) are known. I think the fundamental issue here is that the performance of some CEs fall into the “grey zone” in terms of judging significance of performance (i.e. performance that is not high enough to accept without discussion and not so poor that it is rejected without discussion).

To take a specific example, Huawei experts and experts from other companies have noted that the CE acceptance procedure does not have clear rules as to what is accepted (or rejected). The Chair notes that Huawei has shown an ongoing commitment to the collaborative phase of the standardization process, bringing multiple CEs and also adding to the USAC Common Encoder Reference Software.

In the case of the Huawei Adaptive Time/Filter Post-Processor Filter (ATFPP), Huawei experts feels that the objectives of the ATFPP CE, as expressed by Audio subgroup experts changed over the course of the CE such that it was not possible to clearly know the criterion for success. In the end, it appears that the ATFPP proposal, as a normative postfilter, is not deemed appropriate for MPEG standards in the opinion of some Audio subgroup experts. This was not clear to Huawei experts at the start of the CE proposal phase.

In summary, the Chair stated that he feels very strongly that MPEG has created an excellent standard in USAC, but acknowledges that the collaborative phase, as represented by the CE process, could be be improved.

David Virette, Huawei, stated that the position of Huawei is that the CE process must be improved.