Joint Video Experts Team (jvet) of itu-t sg 6 wp and iso/iec jtc 1/sc 29/wg 11

6.10CE10: Combined and multi-hypothesis prediction (9)

Contributions in this category were discussed Friday 13 July 1600–XXXX (chaired by JRO except otherwise noted)..

JVET-K0030 CE10: Summary report on combined and multi-hypothesis prediction [C.-W. Hsu, M. . Winken, X. . Xiu]
A summary of Core Experiment 10 (CE10) on combined and multi-hypothesis prediction is reported. Four sub CEs are created to test different methods of combined predictions, including CE10.1: multi-hypothesis prediction, CE10.2: overlapped block motion compensation, CE10.3: non-rectangular partitions and CE10.4: diffusion filtering of inter- and intra-prediction signals. In CE10.1, one out of 10 tests was withdrawn and in CE10.2, one out of 3 tests was withdrawn. So there are 9, 2, 3 and 6 tests for each sub CE, respectively. All tests are evaluated based on the common test conditions defined in JVET-J1010. All tests and crosscheck results are integrated in this report.
CE10.1 Multi-hypothesis prediction

In CE10.1, the goal is to test prediction to be combined coming from multiple hypotheses, where one hypothesis refers to prediction from inter mode or from intra mode. The tests and corresponding results are summarized as follows,

#	Proposal	Tester	Supported modes	Hypothesis type	Signalling of hypothesis	# of extra hypothesis	Block constraint in luma samples
CE10.1.1	JVET-K0257	Chih-Wei Hsu (MediaTek)	AMVP uni prediction	inter	merge index	1	>= 8x8
CE10.1.2	JVET-K0257	Chih-Wei Hsu (MediaTek)	skip	inter	implicitly derived	1 or 2
			merge
CE10.1.3	JVET-K0257	Chih-Wei Hsu (MediaTek)	merge	intra	intra mode index	1
CE10.1.4	JVET-K0257	Chih-Wei Hsu (MediaTek)	skip	inter	merge index for uni-prediction + implicitly derived + intra mode index	1 or 2	>= 8x8
			merge	intra
			AMVP uni prediction
CE10.1.5	JVET-K0269	Martin Winken (HHI)	merge	inter	ref index + MVP index + MVDs + weights	1
			AMVP
CE10.1.6	JVET-K0269	Martin Winken (HHI)	merge	inter	ref index + MVP index + MVDs + weights	1 or 2
			AMVP
CE10.1.7	JVET-K0269	Martin Winken (HHI)	merge	inter	ref index + MVP index + MVDs + weights	1	> 8x8
CE10.1.8	JVET-K0269	Martin Winken (HHI)	merge	inter	ref index + MVP index + MVDs + weights	1 or 2	> 8x8
			AMVP
CE10.1.9		withdrawn
CE10.1.10	JVET-K0147	Na Chang (HiSilicon)	merge	inter	implicitly derived	1

By restricting multi-hypothesis to block sizes >=8, the multi-hypothesis prediction does not have worse memory bandwidth requirements than VTM with 4x4.

For each additional hypothesis, another prediction needs to be generated, i.e. the computational complexity would e.g. double in case of uni prediction with 1 additional hypotheses, or bi prediction with 2 additional hypotheses. Each additional hypothesis is then superimposed (with weighted superposition)

The superposition weights are fixed for test 1-4, and can be varied for tests 5-8.

Major differences are:

• 
  Test 1-4 uses fixed weighting 5/8 and 3/8, test 5-8 switches between 2 different weights
  
• 
  Test 1-4 allows combining inter and intra
  

Test 10 generates a second reference for LDP, for which the MV is derived. This requires same number of reference computations as LDB. According to proponents, this performs worse than LDB as such, but has faster encoder.

#	Config.	VTM					BMS
		Y	U	V	EncT	DecT	Y	U	V	EncT	DecT
CE10.1.1	RA	-0.26%	-0.26%	-0.20%	109%	102%	-0.28%	-0.33%	-0.29%	105%	101%
	LB	-0.12%	-0.16%	-0.16%	112%	102%	-0.17%	-0.39%	-0.23%	107%	102%
	LP
CE10.1.2	RA	-0.76%	-0.75%	-0.71%	110%	105%	-0.68%	-0.87%	-0.87%	105%	107%
	LB	-0.40%	-0.38%	-0.52%	115%	104%	-0.51%	-0.67%	-0.73%	109%	102%
	LP
CE10.1.3	RA	-0.68%	-0.68%	-0.51%	112%	104%	-0.62%	-0.30%	-0.33%	104%	102%
	LB	-0.58%	-1.14%	-1.14%	113%	103%	-0.56%	-0.97%	-1.02%	103%	102%
	LP
CE10.1.4	RA	-1.29%	-1.45%	-1.29%	119%	106%	-1.13%	-1.15%	-1.15%	107%	105%
	LB	-0.76%	-1.32%	-1.55%	126%	105%	-0.77%	-1.18%	-1.13%	111%	101%
	LP
CE10.1.5	RA	-1.63%	-1.07%	-1.07%	127%	104%	-0.92%	-0.83%	-0.83%	107%	100%
	LB	-2.05%	-0.39%	-0.44%	145%	107%	-0.93%	-0.47%	-0.39%	114%	99%
	LP
CE10.1.6	RA	-1.91%	-1.24%	-1.19%	130%	104%	-1.07%	-0.91%	-0.95%	108%	100%
	LB	-2.29%	-0.39%	-0.43%	147%	108%	-1.13%	-0.36%	-0.11%	115%	99%
	LP
CE10.1.7	RA	-1.55%	-1.07%	-1.02%	122%	104%	-0.89%	-0.78%	-0.84%	106%	100%
	LB	-2.04%	-0.36%	-0.54%	135%	107%	-0.94%	-0.35%	-0.09%	112%	98%
	LP
CE10.1.8	RA	-1.84%	-1.29%	-1.21%	125%	105%	-1.03%	-0.88%	-0.91%	107%	100%
	LB	-2.27%	-0.32%	-0.48%	138%	108%	-1.07%	-0.29%	-0.12%	114%	100%
	LP
CE10.1.10	RA	0.00%	0.00%	0.00%	101%	100%	0.00%	0.00%	0.00%	100%	100%
	LB	0.00%	0.00%	0.00%	101%	100%	0.00%	0.00%	0.00%	100%	97%
	LP	-3.41%	-2.07%	-1.78%	110%	99%	-1.67%	-1.33%	-1.02%	100%	95%

Question: What would be the effect if only intra and inter are combined? This would be

10.1.3 (combining bi pred and one intra pred), which gives 0.6% gain.

Generally, this experiment provides interesting gain, but requires additional computations (depending on variant), where some of the variants require more memory bandwidth than others. Gain decreases when used in BMS. Should be further studied in combination with other methods of improving motion comp, e.g. improved merge. Further reduction of encoder run time would be desirable as well.