On Constructing z-DimensionalDIBR-Synthesized Images

What is DIBR?

DIBR stands for Depth Image Based Rendering Image-Based Rendering (IBR) is an emerging technology which enables

the synthesis of novel realistic images of a scene from virtual viewpoints, using a collection of available images. The applications of IBR can be found in various situations such as virtual reality, telepresence, thanks to the complexity and performance advantage over model-based techniques, which bases on complex 3-D geometric models, material properties and lightening conditions of the scene

DIBR is IBR technique which maps each color pixel in a reference view to a 2D grid location in the virtual view, using disparity information provided by the corresponding depth pixel.

What is Rendering and Z Dimension?

Rendering is the process of generating an image from a 2D or 3D model (or models in what collectively could be called a scene file) by means of computer programs. 

Three-dimensional space (also: 3-space or, rarely, tri-dimensional space) is a geometric setting in which three values (called parameters) are required to determine the position of an element (i.e., point). This is the informal meaning of the term dimension.

CONSTRUCTION OF Z DIMENSION

To construct this new image type, we first perform a new DIBR pixel-mapping for z-dimensional camera movement.

We then identify expansion holes—a new kind of missing pixels unique in z-dimensional DIBR-mapped images—using a depth layering procedure.

To fill expansion holes we formulate a patch-based maximum a posteriori problem, where the patches are appropriately spaced using diamond tiling.

Leveraging on recent advances in graph signal processing, we define a graph-signal smoothness prior to regularize the inverse problem.

Finally, we design a fast iterative reweighted least square algorithm to solve the posed problem efficiently. Experimental results show that our z-dimensional synthesized images outperform images rendered by a native modification

Virtual Camera  a virtual camera system aims at controlling a camera or a set of

cameras to display a view of a 3D virtual world. Camera systems  where their purpose is to show the action at the best possible angle; more generally, they are used in 3D virtual worlds when a third person view is required.

Constructing z-Dimensional DIBR-Synthesized Images

Divided into three Sections

1.DIBR (Depth Image Based Rendering)2.Image Super Resolution3.Graph Based Image Processing

Depth Image Based Rendering

Color-plus-depth format , consisting of one or more color and depth image pairs from different viewpoints, is a widely used 3D scene representation. Using this format, low-complexity DIBR view synthesis procedure such as 3D warping [ can be used to create credible virtual view images, with the aid of in painting algorithms to complete disocclusion holes

In this work, we assume that enough pixels from one or more reference view(s) have been transmitted to the decoder for virtual view synthesis, and we focus only on the construction of z-dimensional DIBR-synthesized images given received reference view pixels.

Image Super Resolution

Increase in object size due to large z-dimensional virtual camera movement is analogous to increasing the resolution (super-resolution (SR)) of the whole image. However, during z-dimensional camera motion an object closer to the camera increases in size faster than objects farther away, while in SR, resolution is increased uniformly for all spatial regions in the image.

For the above reason, we cannot directly apply conventional image SR techniques [30] in rectangular pixel grid to interpolate the synthesized view. Further, recent non-local SR techniques such as leveraging on self-similarity of natural images that require an exhaustive search of similar patches throughout an image tend to be computationally expensive. In contrast, our interpolation scheme performs only iterative local filtering, and thus is significantly more computation-efficient.

Graph Based Image Processing

GSP is the study of signals that live on structured data kernels described by graphs , leveraging on spectral graph theory for frequency analysis of graph-signals.

Graph-signal priors have been derived for inverse problems such as denoising , interpolation , bit-depth enhancement and de-quantization.

In this work, we assume the latter case and construct a suitable graph G from available DIBR-synthesized pixels for joint denoising/interpolation of pixels in a target patch.

SYSTEM OVERVIEW

Interactive Free Viewpoint Streaming System DIBR Rounding Noise in mapped pixels Identification of expansion holes

Interactive Free Viewpoint Streaming System

Depth-Image-Based Rendering

Rounding Noise in DIBR-Mapped Pixels

Identification of Expansion Holes

CONCLUSION

Unlike typical free viewpoint system that considers only synthesis of virtual views shifted horizontally along the x dimension via DIBR, in this paper we consider in addition construction of z-dimensional DIBR-synthesized images. In such far-to-near viewpoint synthesis, there exists a new type of missing pixels called expansion holes – where objects close to the camera will increase in size and simple pixel-to-pixel mapping in DIBR from reference to virtual view will result in missing pixel areas – that demand a new interpolation scheme.

THANKS

P. Merkle, A. Smolic, K. Mueller, and T. Wiegand, “Multi-view video plus depth representation and coding,

A. Chuchvara, M. Georgiev, and A. Gotchev, “CPU-efficient free view synthesis based on depth layering,” in Proc. 3DTV-Conf: True Vis. - Capture, Transmiss. Display 3D Video, Jul. 2014,

M. Tanimoto, M. P. Tehrani, T. Fujii, and T. Yendo, “Free-viewpoint TV,” IEEE Signal Process. Mag., vol. 28, no. 1, pp. 67–76, Jan. 2011.

REFERENCES