River publishers

Summary

In the family of video coding standards (Fig. 5.1), HEVC has the promise and potential to replace/supplement all the existing standards (MPEG and H.26x series including H.264/AVC). While the complexity of the HEVC encoder is several times that of the H.264/AVC, the decoder complexity [E163] is within the range of the latter. Researchers are exploring about reducing the HEVC encoder complexity [E63, E79, E84, E88, E89, E106, E108, E166 - E17, E187, E195R, E199, E204, E205, E208, E221, E230, E232, E240, E242, E247, E248, E252, E257, E258, E259, E269, E270, 274, E276, E278, E290, E308, E312, E332, E348, E368, E369, E374, E383, E388, E390]. Bin and Cheung [E247] have developed a fast decision algorithm for the HEVC Intra coder and report encoding time is reduced by 52% on average compared with HM 10.0 reference software with negligible losses. Similarly, Kim and Park [E248] developed a fast CU partitioning algorithm resulting in reducing the computational complexity of the HEVC encoder by 53.6% on the average with negligible BD bit rate loss. Chi et al [E246] have developed a SIMD optimization for the entire HEVC decoder resulting in up to 5x speed up. Min, Xu and Cheung [E382} have developed a fully pipelined architecture for intra prediction of HEVC that achieves a throughput of 4 pixels in one clock cycle and is capable to support 3840x2160 video at 30 fps. Correa et al [E383] have developed (part of the abstract is repeated here)

“A set of Pareto-efficient encoding configurations, identified through rate-distortion-complexity

analysis. This method combines a medium-granularity encoding time control with a fine-granularity encoding time control to accurately limit the HEVC encoding time below a

predefined target for each GOP. The encoding time can be kept below a desired target for a wide range of encoding time reductions, e.g., up to 90% in comparison with the original

encoder. The results also show that compression efficiency loss (Bjøntegaard delta-rate) varies from negligible (0.16%) to moderate (9.83%) in the extreme case of 90% computational

complexity reduction”

Index Terms—Encoding time limitation, High Efficiency

In particular, refer to [E242], wherein significant encoder complexity reduction is achieved with negligible increase in BD-rate by combining early termination processes in CTU, PU and RQT structures. See also P.5.139 related to this. Chapter 4 – Intra-picture prediction in HEVC - and Chapter 5 - inter-picture prediction in HEVC - and the references at the end of these chapters [E202] provide insights related to encoder complexity reduction. Another approach in reducing the encoder complexity is based on zero block detection [E166, E171, E186]. Kim et al [E168] have shown that motion estimation (ME) occupies 77-81% of HEVC encoder implementation. Hence the focus has been in reducing the ME complexity [E137, E152, E168, E170, E179, E205, E221, E239, E275, E351, E359, E370, E371]. PPT slides related to fast ME algorithm and architecture design are presented in [Tut10]. Machine learning techniques [E259, E263, E317, E359] have been applied effectively resulting in significant reduction of HEVC encoder complexity. Several researchers have implemented performance comparison of HEVC with other standards such as H.264/AVC (Chapter 4 in [Book4] ), MPEG-4 Part 2 visual, H.262/MPEG-2 Video [E183], H.263, and VP9 (Chapter 6in [Book4] ) and also with image coding standards such as JPEG2000, JPEG-LS, and JPEG-XR [E22, E47, E62, E63, E65, E66, E114, E177, E191, E245]. Similar comparisons with MPEG-4 Part 10 (AVC) and VP9 are reported in [E191]. See also papers listed under VP8, VP and VP10. Most significant of all these is the detailed performance comparison (both objective and subjective) described in [E62]. Special attention may be drawn to references 39, 40, 42-47 cited in [E62]. Chapter 9 - Comparison performance analysis in HEVC – in [E202] provides both objective/subjective comparison between HM 12.1 (HEVC) and JM 18.5 (AVC – H.264). See Table 9.4 in [E202] for description of the test conditions.

However, the payoff is several tests have shown that HEVC provides improved compression efficiency up to 50% bit rate reduction [E61, E62, E99] for the same subjective video quality compared to H.264/AVC (Chapter 4 in [Book4]). See Tables 5.14 and 5.15. Besides addressing all current applications, HEVC is designed and developed to focus on two key issues: increased video resolution - up to 8kx4k – and increased use of parallel processing architecture. A detailed description of the parallelism in HEVC is outlined in Chapter 3 – H. Schwarz, T. Schierl and D. Marpe, “Block structures and parallelism features in HEVC “, in [E202]. This chapter also demonstrates that more than half of the average bit –rate savings compared to H.264 are due to increased flexibility of block partitioning for prediction and transform coding. Brief description of the HEVC is provided. However, for details and implementation, the reader is referred to the JCT-VC documents [E55, E188], overview papers (See section on overview papers).

[E321] is an overview paper describing the range extensions (Rext) of the HEVC. Beyond the Main, Main 10 and Main Intra profiles finalized in version1, Rext (version 2 ) focuses on monochrome, 4:2:4 and 4:4:4 sampling formats and higher bit depths. New coding tools integrated into version 2 are: Cross-component prediction (CCP) [E355], Adaptive chroma quantization parameter offset (ACQP) and residual differential pulse code modulation (RD-PCM). Modification to HEVC Version 1 tools are: Filtering for smoothing of samples in intra-picture prediction, transform skip and transform quantizer bypass modes, truncated Rice binarization, internal accuracy and k-th order Exp-Golomb (EGk) binarization, and CABAC bypass decoding. These tools and modifications are meant to address the application scenarios such as: content production in a digital video broadcasting delivery chain, storage and transmission of video captured by a professional camera, compression of high dynamic range content, improved lossless compression and coding of screen content. This overview paper describes in detail not only the justification for introducing these additional tools, modifications and profiles but also the improved performance. Using a spectrum of standard video test sequences the authors have compared the performance of HEVC version II (HM 16.2 software) with H.264/AVC (JM 18.6) using BD bit rate as a metric. This comparison extends to lossless coding for different temporal structures called as All-Intra (AI), Random Access (RA) and Low Delay (LD). In the overview paper on SCC [E322], implementation complexity as a performance measure is conspicuous by its absence. Extensive list of references related to SCC are cited at the end. See also the overview paper on SCC [SCC48].

Keynote speeches [E23, E99], tutorials (See section on tutorials), panel discussions [E98, E366, E380], poster sessions [E27, E139], workshop [E257] special issues (See special issues on HEVC) test models (TM/HM) [E54], web/ftp sites [E48, E53 - E55], open source software (See section on open source software), test sequences, file formats [E327] and anchor bit streams [E49]. Also researchers are exploring transcoding between HEVC and other standards such as MPEG-2 (See section on transcoders). Further extensions to HEVC [E158] are scalable video coding (SHVC) [E76, E119, E126 - E129, E141, E158, E159, E208, E213, E214, E234, E238, E264 ,E284, E325] (note that E325 is an overview paper on SHVC), 3D video/multiview video coding [E34, E39, E201, E227, E271, E326] (note that E326 is an overview paper on the 3D video/multiview coding) and range extensions [E155, E321] which include screen content coding, See the references on screen content coding besides the overview paper [E322] bit depths larger than 10 bits and color sampling of 4:2:2 and 4:4:4 formats.

In an overview paper, [E325] Boyce et al describe the scalable extensions of the HEVC (SHVC), which include temporal, spatial, quality (SNR), bit depth and color gamut functionalities and as well their combination (hybrid). They compare the performance of SHVC with the scalable extensions of H.264 / AVC called SVC and conclude using various test sequences (Table 5 in [E325]), that SHVC is 50% to 63% more efficient than SVC depending on the scalability type and the use scenario. In view of the superior performance, various standardization bodies such as 3GPP and ATSC for broadcasting and video streaming and IMTC for videoconferencing are evaluating SHVC as potential codec choice. Investigation of bit rate allocation in SHVC is addressed in [E350].

[E322, SCC48] are overview papers on the HEVC SCC extension finalized in 2016. They describe several new modules / tools added to the original HEVC framework and also the new coding tools introduced during the development of HEVC – SCC. [E322] concludes with performance analysis of the new coding tools relative to HEVC – Rext and H.264 / AVC.

Screen content coding (SCC) [E322, SCC48] in general refers to computer generated objects and screen shots from computer applications (both images and videos) and may require lossless coding. HEVC SCC extension reference software is listed in [E256]. Some aspects of SCC are addressed in [E352, E353]. They also provide fertile ground for R & D. Screen content coding has been finalized in 2016. [SCC15] is the special issue on screen content video coding and applications published in Dec. 2016 and reports briefly on ongoing standardization activities. This special issue covers on all aspects of SCC besides opening the avenue for a number of projects/problems. The overview paper on screen content video coding (SCVC) by Peng e al [OP11] addresses recent advances in SCC with an emphasis on two state-of-the-art standards: HEVC/H.265 Screen Content Coding Extensions (HEVC-SCC) by ISO/IEC MPEG and Display Stream Compression (DSC) by VESA. They also explore future research opportunities and standards developments in order to give a comprehensive overview of SCVC. Iguchi et al [E164] have already developed a hardware encoder for super hi-vision (SHV) i.e., ultra HDTV at 7680x4320 pixel resolution. Shishikui [E320] has described the development of SHV broadcasting in Japan and states that 8K SHV broadcasting to begin in 2016 during the Rio de Janeiro Summer Olympics and for full pledged broadcasting to begin in 2020 with the Tokyo Summer Olympics. He has also presented a keynote speech “8K super hi-vision: science, engineering, deployment and expectations”, in IEEE PCS 2015 in Cairns, Australia.

Tsai et al [E222] have designed a 1062 Mpixels/s 8192x4320 HEVC encoder chip. See also [E165] which describes real-time hardware implementation of HEVC encoder for 1080p HD video. NHK is planning SHV experimental broadcasting in 2016. See also [E226]. A 249-Mpixel/s HEVC video decoder chip for 4k Ultra-HD applications has already been developed [E182]. Bross et al [E184] have shown that real time software decoding of 4K (3840x2160) video with HEVC is feasible on current desktop CPUs using four CPU cores. They also state that encoding 4K video in real time on the other hand is a challenge. AceThought among many other companies provides high-quality and efficient software implementation of HEVC Main Profile, Main 10 Profile and Main Still Picture Profile decoder algorithms on various handheld and desktop platforms [E189]. Cho et al [E219] have developed CTU level parallel processing for single core and picture partition level multi core parallel processing for 4K/8K UHD high performance HEVC hardware decoder implementation. They conclude that their multi core decoder design is suitable for 4K/8K UHD real time decoding of HEVC bit streams when implemented on SoC. Kim et al [E220] have extended this to scalable HEVC hardware decoder and the decoder design is estimated to be able to decode 4K@60fps HEVC video. Decoder hardware architecture and encoder hardware architecture for HEVC are addressed in chapters 10 and 11 respectively in the book Sze, Budagavi and Sullivan (editors) “High efficiency video coding (HEVC): algorithms and architectures”, Springer, 2014 [E202]. For 4K video, software decoder has been developed [E329]. A hardware architecture of a CABAC decoder for HEVC that can lead to real time decoding of video bit streams at levels 6.2 (8K UHD at 10 fps) and 5.1 (4K UHD at 60 fps) has been developed by Chen and Sze [E249]. See also [E390]. Zhou et al also have developed similar VLSI architecture for H.265/HEVC CABAC encoder [E250]. Wang et al [E359] have presented an improved and highly efficient VLSI implementation of HEVC MC that supports 7680x4320@60fps bidirectional prediction. The authors [E359] claim that the proposed MC design not only achieves 8x throughput enhancement but also improves hardware efficiency by at least 2.01 times in comparison with prior arts. Zooming and displaying region of interest (ROI) from a high resolution video i.e., ROI based streaming in HEVC has also been developed by Umezaki and Goto [E206]. The authors suggest future work to reduce the decoding cost. A novel Wavefront – Based High parallel Solution for HEVC Encoding integrating data level, optimal single Instruction – Multiple – Data (SIMD) algorithms are designed by Chen et al [E272], in which the overall WHP solution can bring up to 57.65x, 65.55x and 88.17x speedup for HEVC encoding of WVGA, 720p and 1080p standard test sequences while maintaining the same coding performance as with Wavefront Parallel Processing (WPP). Abeydeera et al [E338] have developed a 4K real time decoder on FPGA. Chiang et al {E345] have developed a real-time HEVC decoder capable of decoding 4096x2160@30fps at 270 MHz. Transport of HEVC video over MPEG-2 systems is described in [E197]. Encode/decode and multiplex/demultiplex HEVC video/AAC audio while maintaining lip synch has been implemented in software [E203]. This, however, is limited to HEVC main profile intra coding only. Extension to inter prediction and also to other profiles are valid research areas. Kim, Ho and Kim [E254] have developed a HEVC-complaint perceptual video coding scheme based on JND models that has significant bit-rate reductions with negligible subjective quality loss at a slight increase in encoder complexity. Several companies and research institutes have developed various products related to HEVC (see the booths and exhibits at NAB 2016, held in Las Vegas 16-21 April 2016 www.nabshow.com). One highlight is the single chip #8K #video decoder developed by Socionext US. Several companies have developed or developing hardware encoders/decoders (chips or SOC) that can process up to 4K and 8K video [TUT10]. Details on software codecs are also provided. This tutorial concludes with future challenges for next generation video encoders that relate to extensions such as HSVC, SCC, Multiview, 3D etc. A new standard called MPEG-4 internet video coding (IVC) has been approved as IS0/IEC 14496-33 in June 2015 [E399]. This standard is highly beneficial for video services in the internet domain. The HEVC standard and all its extensions are expected to go through the various revisions and refinements meeting the growth in consumer electronics, internet, VLSI, security, defense, medical, broadcasting, and related areas over the next several years. The next major standard is projected to be around 2020 designed to meet 5G specification. [BH1] and [BH2] address video coding techniques beyond HEVC. References [1] through [8] in [BH2] describe the algorithms and techniques proposed for NGVC beyond HEVC. Sze and Budagavi and also Grois et al (see tutorials) have suggested directions for post HEVC activities. While the main emphasis in developing HEVC has been on efficient video compression and coding, it is interesting to note that the features in its compressed domain have been adopted in video saliency detection [E400].

The projects listed below range from graduate 3 credit hours to research at the M.S. and doctoral levels. These projects are designed to provide additional insights into the tools and techniques and provide a forum for their modifications leading to further improvements in the HEVC encoder/decoder.).Besides the references related to HEVC, references are listed under various categories such as , books, tutorials, transcoders, on line courses (OLC), overview papers, open source software, video test sequences, SSIM, screen content coding, file format, JVT reflector, MSU codec comparison, general, beyond HEVC etc. for easy access. References and projects related to JPEG, JPEG-LS, JPEG-2000, JPEG-XR, JPEG XS, JPEG-XT, PNG, WebP, WebM, DIRAC and DAALA are added at the end. Keynote address by T. Ebrahimi “JPEG PLENO: Towards a new standard for plenoptic image compression”, IEEE DCC, Snow Bird, Utah, 2016 along with the abstract is included. This developing new standard while maintaining backward compatibility with legacy JPEG opens up thought provoking and challenging research [JP-P1]. Panel presentations by Sullivan and others [E366] “Video coding: Recent developments for HEVC and future trends’ during IEEE DCC 2016 opens up research areas for next generation video coding (NGVC). Slides related to high dynamic range, wide color gamut video, 360 video , royalty free video standards are of special interest providing fertile ground for research. Access to HEVC SCC extensions [OP11], new profiles , further compression exploration beyond HEVC is also provided. Several tests [E372] have shown superior performance of HEVC intra compared to JPEG, JPEG LS, JPEG 2000, JPEG XT, JPEG XR and VP9 intra in terms of PSNR and SSIM. Related problems are described in P.5.237 and P.5.238. However, implementation complexity of HEVC intra is higher compared to all the others. Brief introduction to AV1 codec being developed by Alliance for Open Media (AOM) is included. This is followed by brief description of Real Media HD (RMHD) codec developed by Real Networks. Special sessions on recent developments in video coding, genomic compression and quality metrics/perceptual compression to be held in IEEE DCC, April 2017 are of interests to researchers, academia and industry [E386]. Projects related to demosaicking of mosaicked images are listed in P.5.254 thru P.5.257 (See [E384] and [E387]). In IEEE DCC 2017 [E386] special sessions on “Video coding", "Genome Compression", and "Quality Metrics and Perceptual Compression" can lead to number of R&D projects. Of particular interest is perceptual compression which is based on psycho visual perception (vision science). Another interesting topic is keynote address "Video Quality Metrics" by Scott Daly, Senior Member Technical Staff, Dolby Laboratories. Besides compression and coding, some features of HEVC in compressed domain are used in developing video saliency detection [E400]. The authors in [E400] claim that their method is superior to other state-of-the-art saliency detection techniques. Recently, Kim et al [E355] have demonstrated significant coding gains for both natural and screen content video in HEVC based on cross component prediction (CCP). IEEE ICCE, Las Vegas, Jan. 2017 has also many papers related to HEVC. The domain of the video coding standards, in general, is limited to encoder/decoder. Both preprocessing (RGB to color space conversion, quantization and chroma subsampling) and post processing (corresponding inverse processes) are trends for future video codecs. For example HDR and WCG include both pre and post processing. This implies end-to-end pipeline video capture and display in RGB format.

11.