20/11: Work in Progress #Video #Denoising #SuperResolution #AI #DL

Sports video super-resolution and denoising using Deep Learning

(gl) Super-resolución e eliminación de ruído en vídeos deportivos empregando Deep Learning
(es) Super-resolución y eliminación de ruido en vídeos deportivos usando Deep Learning

Student

Oliver José García Sieiro

• Final year project, BSc in Computer Science, UDC
  (TFG Grao en Enxeñería Informática, UDC)

Supervision

Luis Omar Álvarez Mures (Cinfo, UDC)
Francisco Javier Taibo Pena (UDC)
David Maseda Neira (Cinfo)
Emilio José Padrón González (UDC)

Brief description

Super-resolution imaging is the process of enhancing the detail of an image. A low resolution image (LR) is taken as input and it will be upscaled to a higher resolution output (HR). Usually, the input image has a “lower resolution” due to a smaller spatial resolution (i.e. size), due to a result of degradation (such as blurring) or both. We can relate the HR and LR images through the following equation:

LR = degradation(HR)

Clearly, on applying a degradation function, we obtain the LR image from the HR image. But, can we do the inverse? Of course, we usually do not know the degradation function beforehand. Directly estimating the inverse degradation function is an ill-posed problem. In spite of this, Deep Learning (DL) techniques have proven to be effective for super-resolution, with promising results in current research.

Most DL based super-resolution models are trained using Generative Adversarial Networks (GANs). A generative adversarial network (GAN) is a class of machine learning model designed by Ian Goodfellow and his colleagues in 2014 [1]. Two neural networks contest with each other in a game (in the form of a zero-sum game, where one agent’s gain is another agent’s loss).

Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proven useful for semi-supervised learning, fully supervised learning, and reinforcement learning.

In this project we will analyze the current state of the art in DL super-resolution and will experiment with some of the bleeding edge techniques, training a model for our custom dataset and, finally, extending that model to perform denoising as well (i.e. noise reduction/removal). This will allow further testing in a video streaming context, which requires low bandwidth video transmission.

[1] Goodfellow, Ian; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua (2014). Generative Adversarial Networks.

Specific objectives

• Research deep learning super-resolution techniques

• Develop a deep learning model to achieve super-resolution + denoising of an input video stream.

• The specific application domain(s) will be broadcasting, surveillance, etc.

Methodology

An Agile development method will guide the project, with relatively short sprints to build the different tasks, after a preliminary work of study and documentation.

Development steps

• Analysis of requirements and project scheduling, according to student disponibility.

• Study and documentation.

  • GANs.
  • TecoGAN, DRLN…
  • TensorFlow, Pytorch.

• Incremental, iterative work sequences (sprints) to develop and train the chosen model.

• Incremental, iterative work sequences (sprints) to test the video streaming use case.

Material

• Personal computer with GPU and internet access.