International organisation for standardisation

6.11.3Dequantization

6.11.3.1.1.1.1.1.1JCTVC-F257 About clip operation removal from de-quantization part of HM [E. Alshina, A. Alshin (Samsung)]

This contribution proposes a restriction for quantized coefficient values depending on QP and transform size. If the encoder doesn't generate quantized coefficients with a magnitude over this limit then clip operation is not used in the de-quantization part of HM transform. No overflow of 16 bits buffer is guaranteed in this case (provided the encoder behaves as specified).

In principle it is possible to impose normative restrictions on the coefficients, and that would be advantageous to reduce decoder complexity. However there are doubts (see JCTVC-F449 below).

The question is raised whether also cases of potential higher than 10 bits extensions have been considered.

Further study was suggested.

6.11.3.1.1.1.1.1.2JCTVC-F449 Cross-check of Samsung’s contribution on clip operation removal from de-quantization part of HM (JCTVC-F257) [A. Fuldseth (Cisco)]

Comments from the cross-checker:

"Some form of clipping is needed if one wants to limit the dynamic range of the input to the inverse transform, which I believe is a reasonable assumption.

The current quantization/transform design of HM3.0 is so that clipping only occurs in 'extreme' cases:

Near worst case dynamic range of input to the forward transform.

Quantizer that rounds in the 'wrong direction'.

I am unsure whether that situation will ever occur with the current test conditions/sequences.

The proposal moves clipping from after the dequantizer to before the dequantizer which allows for doing clipping in the encoder only. This touches on a more general topic on how to express a limitation on the dynamic range in the HEVC standard:

In the decoding process (e.g. clipping after the dequantizer), or

By imposing constraints on the bitstream (e.g. limitations on the level).

Theoretically one might think that clipping might have slightly less effect if done after the dequantizer (because of higher fidelity), especially at high QP values. I am not sure if that is significant though."