International organisation for standardisation organisation internationale de normalisation

7Explorations – Free Viewpoint Video/Television

The exploratory work on free-viewpoint video has its roots in the “3DAV” exploration, which was originally started in December 2001, and later led in a first CfP on multiview video compression technology (current MVC development in JVT). Based on inputs received and discussions performed in Shenzhen, further clarification was reached on the relationship and potentials of these topics. Considering the MPEG portfolio in stereo and multiview, currently we have / investigate

• 
  23002-3: Defining a format to enable simple stereoscopic application (only one video plus depth possible). In the previous (Lausanne meeting) a request was made to extend 23002-3 by more technology to allow multiple depth layers to better cope with occlusions, but consensus was reached that such approaches should better be subsumed under the FTV perspectives as a possible simple case.
  

• 
  MVC (14496-10/Amd.1): Targets encoding of discrete set of multiple views to highest pixel fidelity, where however current results show that not too much additional compression can be gained by using inter-view similarities; therefore, the necessary rate would still be (depending on number of views) significantly higher than for a single video, while it will be lower than for simulcasting the views. Nevertheless, for high view precision as required in N-view displays, such technology is justified.
  

During the discussion on FTV requirements, it was emphasized that for many consumer-type applications that would require multiview adaptation technology, a large rate overhead (as compared to single view) would be unacceptable. FTV can therefore be defined as a compressed representation and associated technologies which enable generating a large number of different views from a sparse view set. This most probably (from technologies currently known) requires implementation of depth/disparity map estimation (non-normative), definition of depth/disparity map representation/compression, and interpolation/rendering methods (not clear yet whether the latter should be non-normative or normative). All of these elements rely on each other, such that proper technology selection will most probably not be simple. Furthermore, higher distortion may be expected than for MVC (or at least quality may not be measurable in terms of pixel fidelity, and geometric distortions may appear that might only be noticeable under certain observation conditions). The amount of distortion most probably would also depend on compactness (density of views) and complexity of the methods. Depending on the specific application, the view number to be generated may range from two for simple stereoscopic video up to "many" for almost-free walk-through within a scene. Different applications are described in N9466, and it would be desirable to have a common (but scalable) set of technology to accommodate all of them.

General approach:

• 
  N cameras (input views) with only a sparse set of K views encoded
  
• 
  M output views with quality depending on
  
  • 
    Degree of viewing angle X per available input view
    
  • 
    Compactness of representation / data rate R
    
  • 
    Quality of view interpolation
    

• 
  Scalability of technology and backwards compatibility (monoscopic view decodeable) appears important
  

Development of such technology will require careful preparation of a CfP, which could at earliest be issued in July 2008. As a first step, the following two documents were issued:

• 
  FTV Test Cases and Evaluation (N9467)
  
  • 
    Could cover different application needs when only relatively low number of cases are tested (K=1..3)
    
  • 
    Sequences from dense camera settings (from which the relevant views are subsampled) and good depth maps required
    
• 
  Call for Contributions on FTV Test Material (N9468)
  
  • 
    Dense test sequences (specifications given)
    
  • 
    Depth map estimation software (and test cases)
    

Documents reviewed

14876	Andy Tescher for USNB	USNB Contribution: Remarks on multi-view and free-viewpoint video coding work (MVC/FTV) The video subgroup thanks the USNB for the valuable comments on multi-view and free-viewpoint TV coding work. The issue of “inward-looking” and “outward-looking” scenarios is regarded as important, but further study of applications, requirements and prospective technology is needed to identify whether it is useful to establish a common framework for both cases. Regarding the necessity of depth map generation and encoding, a considerable amount of clarification was achieved, for which we refer to N9467 and N9468. The main scope of such technology would be to enable generation of a multitude of views from a much smaller number of encoded viewpoints by utilizing associated depth information. From the applications and requirements study performed in N9466, such a framework could cover a wide range of possible scenarios, which could be seen as unification and extension of technology currently available in ISO/IEC 23002-3 (simple stereoscopic applications) and under development for ISO/IEC 14496-10:200x/Amd.1 (multiview video coding).
14952	Akio Ishikawa, Sei Naito, Shigeyuki Sakazawa	Walk-Through Experience using Ray Space Representation toward Free Viewpoint TV (R)
14949	Tanimoto [e-mail]	Axi-Vision Camera: Real-time depth-mapping HDTV camera
14879	Taka senoh, Kenji Yamamoto, Ryutaro Oi, Tomoyuki Mishina, Makoto Okui	Consideration of FTV image generation from image+depth Depth information can often not be computed correctly. Different cases where actual object distance is needed. Also important for correct rendering of 3D information.
14888	Masayuki Tanimoto, Toshiaki Fujii, Kazuyoshi Suzuki	Multi-view depth map of Rena and Akko&Kayo (R)
14889	Masayuki Tanimoto, Toshiaki Fujii, Kazuyoshi Suzuki	Experiment of view synthesis using multi-view depth (R)
14920	Shinya Shimizu, Hideaki Kimata	View generation from neighboring two videos and two depth maps (R)
14994	Yo-Sung Ho, Sang-Beom Lee, Kwan-Jung Oh, Cheon Lee	Depth Map Generation for FTV
14996	Yo-Sung Ho, Sang-Tae Na, Kwan-Jung Oh, Cheon Lee	Depth Coding and Virtual View Synthesis for FTV

Output documents:

No.	Title	TBP	Available
	Exploration – Free Viewpoint TV Coding
9466	Applications and Requirements of FTV	N	07/10/26
9467	FTV Test Cases and Evaluation	N	07/10/26
9468	Call for Contributions on FTV Test Material	Y	07/11/02