Organisation internationale de normalisation

No.	Title	TBP
Available
	23001-4 Codec Configuration Representation
9984	Study Text of ISO/IEC FCD 23001-4 Codec Configuration Representation	N	08/09/30
	23002-4 Video Tool Library
9985	Study Text of ISO/IEC FCD 23002-4 Video Tool Library	N	08/09/30
9986	WD 2 of ISO/IEC 23002-4/Amd.1 (Conformance and Reference Software)	N	08/07/25
9987	WD 2 of ISO/IEC 23002-4/Amd.2 (Tools for MPEG-4 ASP, AVC HP and SVC)	N	08/07/25
9988	Description of Core Experiments in RVC	N	08/08/08
9989	RVC Work Plan and FU Development Status	N	08/07/25
9990	RVC Vision	Y	08/07/25

4Explorations – 3D Video

The goal of 3D video, as a first step towards a broader range of free-viewpoint (FTV) applications, is to generate interpolated views from available videos of multiview camera configurations. The target application is mostly seen for upcoming generations of (auto-) stereoscopic displays, for which only a low number (1) of video sequences shall be transmitted, but rendering of additional views shall be enabled by associated depth information. To make a next step towards development of such a system, a set of exploration experiments had been set up in Archamps, during which the available test sequences were used with the two available depth estimators and associated view interpolation methods. Various cases of sparseness (baseline distances between cameras to be taken from the dense set) were investigated. Experts viewing using stereoscopic displays was performed in Hannover. Three different display types (two stereoscopic, one autostereoscopic) were available (see N9992 for more details). A first glimpse at the results clearly indicated that only for the “light” case of small baseline distance, results without heavy artifacts could be obtained; therefore, the more intense experts viewing, that involved larger groups of people was only performed to this case. A high-level summary is given here:

a) Results on original views:

Most sequences have been rated “good” for 3D viewing, except “Alt Moabit”, which is an outdoor scene with mostly far-away content. Some comments were made that could be helpful in case of future production of test material:

• 
  Black background is OK when the foreground object has sufficient depth structure (e..g champagne tower)
  
• 
  In general, sequences with more motion / less static background might be desirable (may have implication on encoding and rates to be defined per sequence)
  
• 
  Depth structure was too low in one sequence (dog) – can be resolved by using different camera baseline
  
• 
  Framing problem: Conflict of depth perception when the disparity at screen boundary is large
  

Conclusion: In general, the sequences are suitable for 3D viewing. As a general experience, it is necessary to get familiar with 3D data viewing.

b) Results on synthesized views:

Currently, synthesized views (even for the “light” case) are not acceptable. Main problems are as follows:

• 
  Flickering
  
• 
  Ringing
  
• 
  Ghosting
  
• 
  Temporal Instability
  
• 
  Boundary distortions
  

Conclusion: For the software frame work that is available (Depth Estimation /DE + View Synthesis /VS) the quality is not yet acceptable. It is expected that this situation can be highly improved by relatively simple measures until the next meeting. Before progressing with experiments on AVC/MVC-encoded sequences, further improvements are required for both DE and VS.
c) Judgement of displays:

• 
  Usually larger displays are considered better, but smaller displays had better depth impression sometimes.
  
• 
  It is necessary to clearly define viewing distance, depending on display size, only few people possible simultaneously, controlled seating positions
  
• 
  Stereoscopic vs. autostereoscopic: Stereoscopic has less problems with individual adjustment. This should be the default display, autostereoscopic should also be checked if available, because this is the most likely target of the application (note: the interpolation artifacts are likewise visible on both stereoscopic and autostereoscopic diplays)
  

The next round of experts viewing is planned for the Busan meeting.

The most important step until then is to proceed with improving depth estimation and view synthesis. Those key points will be investigated in a collaborative approach within the next round of Exploration Experiments:

• 
  Improve results of depth estimation (subpel, temporal consistency) – EE 1
  
• 
  Improve results of view synthesis (line projection from rectified images, filling of holes, boundary filtering) – EE 2
  
• 
  Investigate alternative view synthesis from Layered Depth Video (LDV), using same depth maps as from EE 1 – EE 3
  

The EEs will run separately (improving depth estimation and view synthesis) until 2 weeks before next meeting, then the participants will exchange results (software bin or src, depth maps etc.). Combinations of other, newly developed DE (bring depth maps) and VS algorithms can also be computed on-site at next meeting, after AHG reviews the separate results. The Common software basis will be maintained via the MPEG SVN repository.

Once the view synthesis results for the uncoded case provide sufficient quality, encoding experiments (using AVC/MVC both for videos and depth maps) will be performed with the purpose to select appropriate bit rates. This could at earliest start after the Busan meeting. Therefore, the shortest possible tentative timeline could be as follows:

• 
  Until 08/10: Generate improved depth maps and interpolated view sequences from the test sets
  
• 
  08/10: Perform experts viewing with different stereoscopic displays, again decide about maturity of anchors
  
• 
  08/10: Draft CfP (internal)
  
• 
  Until 09/02: Perform experiments to decide about bit rates for the MVC-based video-plus-depth anchors, selection of sequences
  
• 
  09/02: Experts viewing again, preliminary CfP
  
• 
  Until 09/04: Refinement of test conditions
  
• 
  09/04: Final CfP
  
• 
  09/07: Responses due
  

Documents reviewed

m15540	Depth Map Estimation Software version 3	Olgierd Stankiewicz Krzysztof Wegner
m15553	Results of Exploration Experiment in 3D Video Coding for Dog Data Set	Aljoscha Smolic Yongzhe Wang Karsten Mueller
m15573	3D Video Exploration Experiment on Baseline Length Dependency	Taka Senoh Kenji Yamamoto Ryutaro Oi Tomoyuki Mishina Makoto Okui
m15583	Results of Depth Estimation and View Synthesis on Doorflowers Test Data Set for EEs in 3D Video Coding	Shinya Shimizu Hideaki Kimata
m15584	Experimental Results on Depth Estimation and View Synthesis with subpixel-precision	Shinya Shimizu Hideaki Kimata
m15589	Exploration Experiments on seq 'Arrive book' & 'Alt-moabit'	[e-mail] [e-mail]
m15590	Creation of LDV streams out of MV sequences	[e-mail] [e-mail] [e-mail]
m15594	Enhancement of Temporal Consistency for Multi-view Depth Map Estimation	[e-mail] (Sang-Beom Lee) [e-mail] (Yo-Sung Ho)
m15595	Results of Exploration Experiment on View Synthesis	[e-mail] (Cheon Lee) [e-mail] (Yo-Sung Ho)
m15597	Boundary Filtering on Synthesized Images for 3D Video	[e-mail] (Cheon Lee) [e-mail] (Yo-Sung Ho)
m15599	Experimental Results of EE on LeavingLaptop sequence for MPEG-FTV	Gi-Mun Um Gun Bang Namho Hur JinWoong Kim
m15620	Exploration Experiment results on 'Champagne_tower' and 'Pantomime' sequences	Ivana Radulovic Per Fröjdh
m15648	EE results on Leaving laptop	Purvin Pandit Dong Tian Cristina Gomila
m15661	EE results on lovebird2	Jaewon Sung Jung Eun Lim
m15672	View Synthesis Software and assessment of its performance	Mateusz Gotfryd Krzysztof Wegner [e-mail]. Marek Doma?ski
m15674	EE results on lovebird1 sequence	Patrick Lopez Guillaume Boisson
m15696	Simple View Synthesis	Dong Tian Purvin Pandit Cristina Gomila
m15697	Experimental Results for Exploration Experiment in 3D Video	Sehoon Yea Kwan-Jung Oh Anthony Vetro

Output documents: