Open Proposal: 20190218

Feb 18, 2019

Deep Learning Approach for Point-based Rendering

(gl) Unha aproximación Deep Learning para render de nubes de puntos
(es) Una aproximación Deep Learning para render de nubes de puntos

Supervision

Luis Omar Álvarez Mures (Cinfo, UDC)
Francisco Javier Taibo Pena (UDC)
Emilio José Padrón González (UDC)

Brief description

A point cloud is a set of data points in space. Point clouds are generally produced by 3D scanners, which measure a large number of points on the external surfaces of objects around them. As the output of 3D scanning processes, point clouds are used for many purposes, including to create 3D CAD models for manufactured parts, for metrology and quality inspection, and for a multitude of visualization, animation, rendering and mass customization applications [1].

In order to visualize this massive amounts of data one needs to leverage Point-Based Rendering (PBR) techniques [2]. Rendering is the process of generating an image from a model by means of a software program. The model is a description of three dimensional objects in a strictly defined data structure. It contains geometry, viewpoint, texture and lighting information. Rendering is one of the major field of 3D computer graphics.

Using points as the rendering primitive, output images are constructed from a cloud of points and this is known as Point-Based Rendering (PBR). Comparing polygon meshes with points is analogous to comparing vector graphics with pixel graphics. Points in 3D are analogous to pixels in 2D, replacing textured triangles or higher order surfaces by zero-dimensional elements.

Since points are adimensional, they need to be used to calculate normals and use them to provide the visual representation with a surface, which is then called Splat. The process of estimating normals is non-trivial. Point sets are usually quite noisy and have no more information than an (x,y,z) point position.

Deep learning [3] architectures such as deep neural networks, deep belief networks and recurrent neural networks have been applied to fields including computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, bioinformatics, drug design, medical image analysis, material inspection and board game programs, where they have produced results comparable to and in some cases superior to human experts.

Deep learning models are vaguely inspired by information processing and communication patterns in biological nervous systems yet have various differences from the structural and functional properties of biological brains (especially human brains), which make them incompatible with neuroscience evidences.

The goal of this project will be the implementation of state-of-the-art PBR techniques and normal estimation algorithms. Deep Learning techniques will then be tested out to try to improve on these classical approaches. In order to achieve these results, OpenGL will be used to render said primitives in real-time. On the Deep Learning side, TensorFlow will be used since it is the defacto standard for Machine Learning and Deep Learning models.

The specific application domains are 3D modeling, civil engineering, architecture, etc.

[1] A Overview of Point-based Rendering Techniques.
Mandakini Kaushik, Kapil Kumar Nagwanshi, Lokesh Kumar Sharma.
International Journal of Computer Trends and Technology (IJCTT),
V3(1):19-24. 2012. ISSN 2231-2803.
http://www.ijcttjournal.org.

[2] Point-Based Graphics.
Markus Gross, Hanspeter Pfister.
Morgan Kaufmann. 2007.
ISBN: 0123706041 9780080548821

[3] Deep Learning in Neural Networks: An Overview
Juergen Schmidhuber.
Neural Networks, Vol 61, pp 85-117, Jan 2015
DOI: 10.1016/j.neunet.2014.09.003

Specific objectives

  • The main objective of this project is to develop PBR and normal estimation techniques with OpenGL and TensorFlow.

  • These techniques will explore new innovative Deep Learning methods.

  • A standalone proof-of-concept application will be developed.

Methodology

An Agile development method will guide the project, with relatively short sprints to build the different tasks, after a preliminary work of study and documentation.

Development steps

  • Analysis of requirements and project scheduling, according to student disponibility.

  • Study and documentation.

    • PBR techniques and normal estimation.
    • C++, OpenGL and TensorFlow.

  • Incremental, iterative work sequences (sprints) to develop classical PBR and normal estimation techniques.

  • Incremental, iterative work sequences (sprints) to develop Deep Learning approach for normal estimation.

Material

  • Personal computer with GPU and internet access.
Computer Graphics Point-based Rendering Interactive Rendering Deep Learning AI GPU
Teaching and Researching in Computer Science/Engineering

My research interests include High Performance Computing (HPC) and Computer Graphics.