Joint Collaborative Team on Video Coding (jct-vc)

6.6SEI messages

6.6.1Motion and prediction constrained SEI messages

JCTVC-N0069 Motion and inter-layer prediction constrained SEI message [K. Ugur, M. M. Hannuksela (Nokia)]

(Reviewed Thu. 25th p.m. in Track A (GJS).)

One potential application of SHVC is to enhance the quality of a region of interest (ROI) of the base layer picture with the enhancement layer. In such use, as only a certain region of the picture is enhanced, it would be useful to indicate the details of this operation to avoid redundant decoding of certain regions and minimize complexity. It is asserted that tiles, and more specifically motion-constrained tiles, can be used to support the ROI quality enhancement use case. This contribution proposes some modifications to the motion-constrained tile sets SEI message with the aim of supporting ROI enhancement.

Two scenarios:

• 
  Enhancing an ROI, leaving some non-ROI area the same as the base layer (except perhaps for resampling).
  
• 
  Enhancing an ROI without using BL texture, leaving some non-ROI area the same as the base layer. (A further constraint relative to the first scenario.)
  

The second scenario is related to N0117 and N0236.

Proposes, by adding additional syntax elements for each tile set rectangle, to:

• 
  Indicate "skipped tile sets" where the EL area is identical to the (possibly upsamplesd) BL.
  
• 
  Indicate no-inter-layer-prediction tile sets.
  

In discussion, it was asked what is the purpose of the layer ID syntax element and its relationship with scalable nesting and the layer ID in the NUH.

It was remarked that N0159 is also related.

See BoG report N0374 and associated notes.

JCTVC-N0087 Inter-layer constrained tile sets SEI message [K. Suehring, R. Skupin, Y. Sanchez, T. Schierl (FhG HHI)]

(Reviewed in Track B Sun. 28th (AS).)

This contribution proposes an SEI message that indicates inter-layer decoding independence between tiles. The syntax is aligned with the (temporal) motion constrained tile sets SEI message. In addition to region of interest based decoding, a decoder can use the information provided by the SEI message for load assignment, e.g. parallel decoding of tiles.

Uses the structure of the motion constrained tile sets SEI message to signal that tiles are independent for inter-layer prediction.

In addition, the proposal also contains a flag to signal that all the tiles are independent.

This is similar to N0159 (proposal 3).

The proposal is also applicable to multi-view coding.

It was commented that there may be a conflict between this application and region of interest signalling.

Others asserted that the proposed method could co-exist with the signalling of region of interest tile sets.

It was commented that region of interest and parallelisation, which is the goal of this proposal, are different applications.

One expert commented that the relationship between motion constrained tile sets and inter-layer constrained tile sets should be discussed.

See also BoG report N0374 and related notes.

JCTVC-N0088 Extension of (temporal) motion constrained tile sets SEI message [K. Suehring, R. Skupin, Y. Sanchez, T. Schierl (FhG HHI)]

See BoG report N0374 and related notes.

JCTVC-N0117 HLS: Extensions to Motion-constrained tile sets SEI message [S. Hattori, O. Nakagami, T. Suzuki (Sony)]

See BoG report N0374 and related notes.

JCTVC-N0236 HLS: Extensions of motion-constrained tile sets SEI message for interactivity [C. Auyeung, J. Xu (Sony)]

See BoG report N0374 and related notes.

6.6.2Layer presence and dependency change SEI messages

JCTVC-N0173 MV-HEVC/SHVC HLS: On Layers Present SEI message [J. W. Kang (ETRI), J. Lee, H. Lee, J. S. Choi, T. C. Thang (UoA)]

See BoG report N0374 and related notes.

JCTVC-N0174 MV-HEVC/SHVC HLS: On Layer Dependency Change SEI message [T. C. Thang (UoA), J. W. Kang, J. Lee, H. Lee, J. S. Choi (ETRI)]

See BoG report N0374 and related notes.

6.6.3Frame packing SEI messages

JCTVC-N0133 HLS: Frame packing arrangement SEI message for bit depth extension [B. Choi, Y. Cho, M. W. Park, J. Y. Lee, H. Wey, C. Kim (Samsung)] [late]

A change of the frame packing arrangement (FPA) SEI message is proposed for delivering extended bit-depth data up to 14 bits in nominally 8bit bitstreams. Two layered bitstreams are carried with FPA SEI messages. One constituent picture can be decoded compatibly as an ordinary 8 bit image, or can be decoded an extended bits (10~14bits) image with a supplementary picture data, which are differential data between the original data and the coded 8 bits data, in a scalable manner.

Further study was encouraged.

JCTVC-N0134 HLS: Frame packing arrangement SEI message for full resolution 3D [B. Choi, Y. Cho, M. W. Park, J. Y. Lee, H. Wey, C. Kim (Samsung)] [late]

A change of the frame packing arrangement (FPA) SEI message is proposed. The proposal is to add a flag, which indicates whether the current video sequence is the "primary half resolution 3D data" or is complementary (secondary) data. It is asserted that the full resolution 3D video as well as the half resolution 3D video can be delivered in a scalable manner with the proposed FPA SEI message.

Further study was encouraged.

JCTVC-N0270 Updated proposal with software for frame packing arrangement SEI message for 4:4:4 content in 4:2:0 bitstreams [Y. Zhang, Y. Wu, S. Kanumuri, S. Sadhwani, G. J. Sullivan, H. S. Malvar (Microsoft)]

(Reviewed Fri. 2nd a.m. (JRO).)

This contribution proposes a method to extend the use of the frame packing arrangement SEI message to represent 4:4:4 content in nominally 4:2:0 bitstreams. The contribution is an update of the prior contributions JCTVC-K0240, JCTVC-L0316 and JCTVC-M0281 that provides software and specification text with a graphical illustration (as reported to be requested during discussion at the previous meeting). With the proposed method, it is reported that one constituent frame (e.g. in a top-bottom packing or alternating-frame coding scheme) can be decoded compatibly as an ordinary 4:2:0 image, or can be supplemented with the data from another constituent frame to form a complete 4:4:4 image representation. It is proposed to include support for the additional scheme into the frame packing arrangement SEI message in both AVC and HEVC, to facilitate deployment of systems using this method. Since 4:2:0 is the most widely supported format in products, it is asserted that having an effective way of conveying 4:4:4 content through such decoders can provide the substantial benefit of enabling widespread near-term deployment of 4:4:4 capabilities (especially for screen content coding). The proposed method operates by packing the samples of a 4:4:4 frame into two 4:2:0 frames and encoding the two 4:2:0 frames as the constituent frames of a frame packing arrangement. The semantics of 'content_interpretation_type' are extended to signal this packing arrangement. The proposed scheme is asserted to be of high practical value for applications involving screen content. Relative to native 4:4:4 encoding, the proposed scheme can provide the advantage of compatibility with the ordinary 4:2:0 decoding process that is expected to be more widely supported in decoding products. It is reported that the attached software is capable of handling the frame-packing and frame-unpacking processes and can be used in conjunction with any 4:2:0 codec.

Basically the same as previously proposed. Update includes software and improved text.

Three types of 4:2:0 low-resolution (new interpretation types 3, 4, and 5):

without filtering, two informatively describing downsampling and reconstruction: conventional filters; lifting based. Phase position as from annex E.

Question: Necessary to specify the filters precisely? Lifting filters could be implemented via conventional?

Lifting filters would allow (almost) perfect reconstruction (for PR, it would require higher bit depth)

One expert supports the proposal.

Using existing frame packing, but adding interpretation type.

The group initially agreed to adopt this for the SHVC draft, but only interpretation types 3 and 4, without specifying anti alias and upsampling filters.

However, after further discussion, it was agreed to defer the planned action to consideration at the next meeting.

6.6.4Color enhancement SEI messages

JCTVC-N0148 AhG8: Guided Image Filtering for Screen Content Coding [T. Vermeir (Barco), J. De Cock, G. Van Wallendael, S. Van Leuven (Ghent Univ. - iMinds)]

(Reviewed Thu 1st a.m. (GJS).)

Downsampling the chroma components, as used for coding in HEVC Main profile (4:2:0), generates visual artifacts for screen content. This contribution provides the results of experiments with a guided image post-processing filter to reduce these artifacts after decoding. This filtering step is asserted to restore edges in the chroma component of screen-content, which have been affected due to the original 4:2:0 sub-sampling. The luma component is used as a guide to enhance the chroma components. By analyzing a local histogram per pixel, the radius of the guided image filter is reconfigured. Tests conducted on several sequences reportedly show that the PSNR of the Cb and Cr components for lossless compressed streams is increased by 0.63 dB and 0.99 dB on average, respectively.

The presented results are based on on-going experiments, therefore also future work is proposed.

This is described as a post-process, not something that happens within the prediction loop.

The subjective quality improvement is suggested to exceed the measured PSNR benefit.

Hypothetically, this could be supported using an SEI message.

The contributor suggested that this could also be used in chroma format scalability.

The reported quality improvement seems substantial.

Further study was encouraged.

JCTVC-N0180 Color Mapping SEI message [Philippe Bordes, Pierre Andrivon, Patrick Lopez, Franck Hiron (Technicolor)]

This contribution proposes a colour mapping SEI message to support colour mapping post-processing on reconstructed frames. The colour mapping function is coded using a 3D Colour LUT.

It was remarked that this seems closely related to new parent-level inputs under substantial consideration – e.g. input from studios on XYZ and input submitted for CICP consideration.

It may also be related to the colour gamut scalability, in the sense that it provides a way to provide compatible service targeting alternative displays.

It was commented that VUI (SPS or VPS VUI) could be considered here as an alternative way of expressing the information. The contributor suggested that this might be too much data to carry at the VPS level. It was further remarked that perhaps there could be a combination of some small indicator at a high level with details provided at a lower syntax level.

Further study was encouraged.

JCTVC-N0224 SEI message: post filters to enhance the chroma planes [J. Dong, Y. Ye, Y. He (InterDigital)]

This contribution proposes an SEI message carrying post filter coefficients, in order to enhance the quality of the reconstructed chroma planes. On the decoder side, a reconstructed chroma pixel is enhanced by adding an appropriate offset obtained by applying the received post filters, which usually have high-pass characteristics, on the surrounding luma pixels. By doing this, the chroma edges lost during compression are well restored using the high frequency components from the corresponding luma plane. The proposed SEI message is implemented in HM-10.1 and HEVC RExt. software “HM-10.1+RExt-3.0”. Compared with HM-10.1 anchor and HEVC RExt. anchor, significant chroma PSNR improvement, e.g., up to 1.5 dB, can be observed in a wide range of sequences, including Class A, Class B, screen contest sequences, and YUV sequences with higher bit-depth (10-bit) and color sampling ratios (4:4:4 and 4:2:2).

It was noted that N0224 is a post-filter approach corresponding to the same idea described in N0223.

Some variant was also considered in the SHVC context in JCTVC-M0183 and in a CE contribution to this meeting.

We also have the post-filter hint SEI message.

This is in the area of post-filtering enhancement, and hypothetically we know there are other things we could do, e.g. ALF as a post-filter with block-level adaptivity. We have not had other recent inputs on post-filtering enhancement information.

Significant PSNR gain in the chroma domain was reported (up to 1.5 dB, esp. at lower bit rates in the range of 39 dB PSNR).

It was asked whether a post-filter scheme that does not have the cross-component characteristic would provide similar amounts of gain. The contributor responded that the high-pass nature of this filter might provide visually superior results.

It was remarked that there is interest in defining post-filtering enhancement information and that it would be desirable to study the general area in order to try to identify the best possible schemes for such technology.

Further study was encouraged for the scheme, and for other post-filtering methods as well.

JCTVC-N0309 Signalling of chroma sampling filter [K. Kazui (Fujitsu), T. Chujoh (Toshiba)] [late]

JCTVC-M0146 provided a framework to send recommended up-sampling and down-sampling filter coefficients for chroma format conversion by using VUI and SEI message. This follow-up contribution proposes a modified signalling scheme of chroma sampling filter taking into consideration the discussion on JCTVC-M0146 from the last meeting.

Two variations were provided – VUI or SEI.

A first contribution was submitted in October on the subject.

Only one specific type of filtering indication was SMPTE RP2050-1 (with guidelines in SMPTE EG2050) for vertical and JPEG 2000 5/3. It was suggested that providing the JPEG 2000 values would not take much text and would avoid a need to find the information elsewhere.

Regarding rounding, the prop refers to SMPTE EG, and it was suggested that there is rounding advice in JPEG 2000.

Suggested to be sent (only) when the encoding is 4:2:0 or 4:2:2. This should be clarified.

Editorial improvement seems needed for the draft text.

SEI rather than VUI suggested, e.g. due to the size.

The amount of text is substantial and had not been reviewed in detail.

It was agreed to plan to adopt at the next meeting, given adequate editorial improvement, good candidate reference software being provided (perhaps with external tool, probably usable with HM), and no new issues discovered.

6.6.5Other SEI messages

JCTVC-N0286 Time code SEI message [C. Fogg, B. Lewis, J. Lassahn, T. Prins (Elmental)]

This proposal requests that some clock_timestamp elements from AVC be copied into a new HEVC high level syntax construct, such as an SEI message, that can accompany each coded picture in the elementary video bitstream. Time code is informative, and has no impact on the decoding, HRD, or any output processes within the scope of the HEVC specification. Although several user_data() conventions for time code are used in exchange within specific applications, the MPEG-2 GOP header and AVC pic_timing() SEI message have reportedly served as universal embedded time codes that are relied upon in practice by several encoders across applications to synthesize systems timing information, in particular the PTS and DASH timeline. Time code is also passed through the elementary stream for various uses downstream. The authors of this proposal suggest that, especially for the benefit of transcoding operations from AVC to HEVC, a simplified SEI time code be re-introduced into the next HEVC draft. The most essential information is suggested to be the {Hours, Minutes, Seconds, Picture Count} parameters commonly known as “SMPTE 12M”. An alternative syntax adapted from AVC is also provided that includes more of its complicated features, should some unknown users rely on drop frame counts and/or need more than 24 hours of time range indication. It is proposed that time code be signalled only for coded pictures, and therefore any repeat pictures (indicated by pic_struct and DeltaToDivisor in HEVC Table E-6) be implicitly derived, rather than explicitly signalled as is done via the AVC clock_timestamp[i] array.

The presenter was not available Fri 2nd a.m. The contribution is available for study.