Hi there! And welcome to my website!

My name is Emilio José Padrón González (Ourense, 1975) and I am teaching and researching at Universidade da Coruña (UDC) as a member of the Computer Architecture Group (GAC) and the research center CITIC. My main topics of interest are in the fields of High Performance Computing (HPC), Computer Graphics, Data Analytics and Artificial Intelligence.

At present, I’m mainly focus on these two research lines:

  • High performance data analytics (HPDA and Big Data), applied to scientific data from numerical simulations to get an online analysis of these data (in-situ and/or in-transit) at scale.

  • The triangle Computer Graphics - Computer Vision - Artificial Intelligence: particularly, IA (ML/DL/DRL) applied to computer graphics, computer vision and multimedia.

Interests

  • High Peformance Computing
  • Computer Graphics
  • Data Analytics
  • Artificial Intelligence
  • Free/Libre and Open-Source Software (FLOSS)

Education

  • PhD in Computer Science, 2006

    Universidade da Coruña

  • MSc in Computer Science, 1998

    Universidade da Coruña

  • BSc in Computer Science, 1996

    Universidade da Coruña

Teaching

2020-2021 Academic year

Degree: Bachelor of Computer Science / Grao en Enxeñería Informática

Courses:

Degree: Master of Computer Science / Mestrado en Enxeñería Informática (MUEI)

Course:

  • Interaction, Graphics and Multimedia / Interacción, Gráficos e Multimedia (IGM)
    -> File & teaching guide

Degree: Master in HPC (High Performance Computing) / Mestrado en Computación de Altas Prestacións

Course:

Degree: Master in Design, Development and Marketing of Videogames / Mestrado en Deseño, Desenvolvemento e Comercialización de Videoxogos

Course:

  • Video Game Performance and Optimization / Rendemento e optimización de videoxogos
    -> File & teaching guide

Final Year Project Supervision: BSc & MSc

Call for proposals!

21/02: Work in Progress #Agriculture #AI #RL

Crop Models for Reinforcement Learning.

21/02: Work in Progress #HPDA #Python #RAY

Online Data Analytics for Scientic Numerical Simulations with Python & Ray.

20/11: Work in Progress #Video #Denoising #SuperResolution #AI #DL

Sports video super-resolution and denoising using Deep Learning.

Archive

Past supervised projects.

20/06: Work in Progress #Rendering #Points #AI #DL

Deep Learning Approach for Point-based Rendering.

20/03: Work in Progress #Video #AI #DL

Deep Learning chroma keyer implementation.

Recent Publications

Quickly discover relevant content by filtering publications.

Contact

  • +34-881016085
  • D3.15, Edificio Área Científica, Campus Elviña s/n, A Coruña, 15071
  • Tuesday 10:30 to 13:30
    Thursday 15:30 to 17:30
    Friday 10:30 to 13:30
    Otherwise email to book an appointment
  • DM Me
  • Telegram Me

Teaching

2020-2021 Academic year

Degree: Bachelor of Computer Science / Grao en Enxeñería Informática

Degree: Master of Computer Science / Mestrado en Enxeñería Informática (MUEI)

  • Course:

    • Interaction, Graphics and Multimedia / Interacción, Gráficos e Multimedia (IGM)
      -> File & teaching guide

Degree: Master in HPC (High Performance Computing) / Mestrado en Computación de Altas Prestacións

Degree: Master in Design, Development and Marketing of Videogames / Mestrado en Deseño, Desenvolvemento e Comercialización de Videoxogos

  • Course:

    • Video Game Performance and Optimization / Rendemento e optimización de videoxogos
      -> File & teaching guide

PFCs

Archive

Past EOG Works and Final Master Thesis supervised (en)
Histórico de Traballos Fin de Grao e de Mestrado dirixidos (gl)

2020

  • GPU-based mapping of multispectral images on huge 3D point clouds :MHPC:

    Mapeo de imaxes multiespectrais sobre nubes de puntos 3D de alta densidad en GPU (gl)
    Mapeo de imágenes multiespectrales sobre nubes de puntos 3D de alta densidad en GPU (es)
    by Juan Manuel Jurado Rodríguez
    co-advised with Francisco Ramón Feito Higueruela (Universidad de Jaén)
    -> pdf (es) -> slides (es)

  • Reinforcement Learning multiagent system for simulating a survival environment :MUEI:

    Sistema multiaxente de aprendizaxe por reforzo para simulación de contorna de supervivencia (gl)
    Sistema multiagente de aprendizaje por refuerzo para simulación de entorno de supervivencia (es)
    by Rubén Montero Vázquez
    co-advised with Luis Omar Álvarez Mures (Cinfo, UDC) and Francisco Javier Taibo Pena (UDC)
    -> pdf (en)
    -> https://gitlab.com/ruben.montero/town-survival-rl-simulator

2018

  • Design and implementation of a videogame in the field of Human Rights based on free software tools :MUEI:

    Deseño e implementación dun videoxogo no ámbito dos dereitos humanos baseado en ferramentas libres (gl)
    Diseño e implementación de un videojuego en el ámbito de los derechos humanos basado en herramientas libres (es)
    by Daniel José Barbeira Hayes
    co-advised with Miguel Rodríguez-Segade Alonso (ESF, Enseñería sen Fronteiras) and Sergio Fernández Alonso (ESF, Enseñería sen Fronteiras)
    -> pdf (gl)
    -> https://gitlab.com/daniel.barbeira/blockrain.js

  • In-Transit Molecular Dynamics Analysis with Apache Flink :Grenoble:

    Master of Science in Informatics, specialization Parallel, Distributed and Embedded Systems. Université Grenoble Alpes.
    by Henrique Colao Zanúz
    co-advised with Bruno Raffin (Inria Rhône-Alpes, Datamove team)
    -> pdf (en)

  • Interactive voxel-based rendering engine: study and development :GEI:

    Estudo e desenvolvemento dun motor gráfico interactivo baseado en voxels (gl)
    Estudio y desarrollo de un motor gráfico interactivo basado en voxels (es)
    by Víctor Castro Cabado
    co-advised with Diego Andrade Canosa (UDC, Dpt. Enxeñería de Computadores)
    -> http://kmelot.biblioteca.udc.es/record=b1662870~S1*gag
    -> pdf (es)

2016

  • Development of a web application for historical investigation and dissemination :GEI:

    Desenvolvemento dunha aplicación web para a investigación e divulgación histórica (gl)
    Desarrollo de una aplicación web para la investigación y divultación histórica (es)
    by José Enrique Benlloch Castiñeira
    -> http://kmelot.biblioteca.udc.es/record=b1541529~S1*gag

  • Planar surfaces recognition using genetic algorithms in CUDA :ETIS:

    Recoñecemento de superficies planas con algoritmos xenéticos en CUDA (gl)
    Reconocimiento de superficies planas con algoritmos genéticos en CUDA (es)
    by Iván Lago Castro
    -> http://kmelot.biblioteca.udc.es/record=b1544003~S1*gag

  • Deployment and integration of a collaborative email platform with a storage system in a private cloud :ETIX:

    Implantación e integración dunha plataforma de correo colaborativo cun sistema de almacenamento nunha nube privada (gl)
    Implantación e integración de una plataforma de correo colaborativo con un sistema de almacenamiento en una nube privada (es)
    by Lorenzo Rodríguez Pérez
    -> http://kmelot.biblioteca.udc.es/record=b1543792~S1*gag

  • Web application to manage workflows within a company :ETIX:

    Aplicación web para a xestión do fluxo de traballo dentro dunha empresa (gl)
    Aplicación web para la gestión del flujo de trabajo dentro de una empresa (es)
    by Marcos Pérez Vázquez

  • Application for managing a country store :ETIX:

    Desenvolvemento dunha aplicación para a xestión dunha tenda de ultramarinos no rural (gl)
    Desarrollo de una aplicación para la gestión de una tienda de ultramarinos rural (es)
    by Gerardo Manuel López Fernández
    -> http://kmelot.biblioteca.udc.es/record=b1543790~S1*gag

2015

  • Design and implementation of a HPC solution for Stereo Matching :MICAP:

    Deseño e implementación dunha solución HPC para Stereo Matching (gl)
    Diseño e implementación de una solución HPC para Stero Matching (es)
    by Luis Omar Álvarez Mures
    co-advised with Juan Ramón Rabuñal Dopico (UDC, Dpt. Computación)
    -> pdf (en)
    -> https://github.com/omaralvarez/GCVL

  • Design and implementation of the deployment in the cloud of a highly-­scalable platform for multimedia metadata

    Deseño e implementación do despregamento na nube dunha plataforma altamente escalable de metadatos multimedia (gl)
    Diseño e implementación del despliegue en la nube de una plataforma altamente escalable de metadatos multimedia (es)
    by Juan Font Alonso
    co-advised with Laura Milagros Castro Souto (UDC, Dpt. Computación)
    -> pdf (gl)

  • Implementation of a stock management system for an e-commerce platform :GEI:

    Implementación dun sistema de xestión de stock para unha plataforma de comercio electrónico (gl)
    Implementación de un sistema de gestión de stock para una plataforma de comercio electrónico (es)
    by Juan Manuel García Sánchez
    -> http://kmelot.biblioteca.udc.es/record=b1531121~S1*gag

2014

  • Implementation of ITILv3 in the IT services of a local government :GEI:

    Implantación de ITILv3 nos servizos TI dunha administración local (gl)
    Implantación de ITILv3 en los servicios TI de una administración local (gl)
    by Gloria Picos Sedes
    co-advised with Ramón Álvarez Veiras (Concello da Coruña, Dpt. Informática)

  • Real-time management tool massive 3D point clouds :GEI:

    Ferramenta para o traballo interactivo con grandes nubes de puntos 3D (gl)
    Herramienta para el trabajo interactivo con grandes nubes de puntos 3D (es)
    by Luis Omar Álvarez Mures
    co-advised with Alberto Jaspe Villanueva (Italy, CRS4, Visual Computing Group)
    -> http://kmelot.biblioteca.udc.es/record=b1521392~S1*gag

2013

  • Real-time multiresolution 3D visualization system for huge LIDAR datasets :GEI:

    Sistema multirresolución de visualización 3D en tempo real para grandes bases de datos LIDAR (gl)
    Sistema multirresolución de visualización 3D en tiempo real para grandes bases de datos LIDAR (es)
    by Alberto Jaspe Villanueva
    -> http://kmelot.biblioteca.udc.es/record=b1515710~S1*gag
    -> pdf (es)

  • Conjugate gradient for sparse matrices on GPU using CUDA :EI:

    Gradiente conxugado para matrices dispersas sobre GPUs usando CUDA (gl)
    Gradiente conjugado para matrices dispersas sobre GPUs usando CUDA (es)
    by María del Carmen Pena Lourés
    co-advised with Margarita Amor López (UDC, Dpt. Enxeñería de Computadores)
    -> http://kmelot.biblioteca.udc.es/record=b1506243~S1*gag

  • Client-server point-based rendering WebGL :EI:

    Sistema cliente-servidor para a visualización de nubes de puntos con WebGL (gl)
    Sistema cliente-servidor para la visualización de nubes de puntos con WebGL (es)
    by Javier Rey Neira
    co-advised with Alberto Jaspe Villanueva (Italy, CRS4, Visual Computing Group)
    -> http://kmelot.biblioteca.udc.es/record=b1506989~S1*gag

  • Software for managing the budget, planning and cost control in a construction company :GEI:

    Aplicación para a xestión de orzamentos, planificación e control de custos de obra dunha empresa de construción (gl)
    Aplicación para la gestión de presuspuestos, planificación y control de costes de obra de una empresa de construcción (es)
    by Raúl Fernández Núñez
    -> http://kmelot.biblioteca.udc.es/record=b1516169~S1*gag

2012

  • Point Cloud Manager: A multi-resolution framework for managing huge 3D point cloud datasets :MICAP:

    Point Cloud Manager: Sistema Multirresolución para o tratamento de grandes datasets de nubes de puntos 3D (gl)
    Point Cloud Manager: Sistema Multirresolución para el tratamiento de grandes datasets de nubes de puntos 3D (es)
    by Alberto Jaspe Villanueva
    co-advised with Javier Taibo Pena (UDC, Dpt. Enxeñería Civil)
    -> pdf (es)

  • Global illumination for point-based rendering :ETIS:

    Iluminación global para render baseado en puntos (gl)
    Iluminación global para render basado en puntos (es)
    by Luis Omar Álvarez Mures
    co-advised with Alberto Jaspe Villanueva (Italy, CRS4, Visual Computing Group)
    -> http://kmelot.biblioteca.udc.es/record=b1488553~S1*gag

  • Building a robot to autonomously traverse the whole area inside a perimeter :ETIS:

    Construción dun robot que de xeito autónomo recorra toda a área delimitada por un perímetro (gl)
    Construcción de un robot que de forma autónoma recorra toda el área delimitada por un perímetro (es)
    -> http://kmelot.biblioteca.udc.es/record=b1490803~S1*gag

  • A tool for the creation and management of basketball moves and exercises :ETIS:

    Ferramenta para a creación e xestión de xogadas e exercicios de baloncesto (gl)
    Herramienta para la creación y gestión de jugadas y ejercicios de baloncesto (es)
    -> http://kmelot.biblioteca.udc.es/record=b1490808~S1*gag

2011

  • Performance analysis of CUDA 4.0 for multiple GPUs :EI:

    Análise de rendemento multiGPU en CUDA 4.0 (gl)
    Análisis de rendimiento multiGPU en CUDA 4.0 (es)
    co-advised with Margarita Amor López (UDC, Dpt. Enxeñería de Computadores)
    -> http://kmelot.biblioteca.udc.es/record=b1473774~S1*gag

  • Design of a VPN-based deployment architecture with centralized configuration upon Pulsarent’s requirements :ETIX:

    Deseño dunha arquitectura de despregue baseada en VPN e de configuración centralizada supeditada aos requisitos do produto de cartelería dixital PulsarRent (gl)
    Diseño de una arquitectura de despliegue basada en VPN y de configuración centralizada supeditada a los requisitos del producto de cartelería digital PulsarRent (es)
    -> http://kmelot.biblioteca.udc.es/record=b1469501~S1*gag

2010

  • Parallel computation in a CPUs-GPUs heterogenerous environment :ETIS:

    Computación paralela nunha contorna heteroxénea CPUs-GPUs (gl)
    Computación paralela en un entorno heterogéneo CPUs-GPUs (es)
    co-advised with Bruno Raffin (Inria Rhône-Alpes, Datamove team)
    -> http://kmelot.biblioteca.udc.es/record=b1460801~S1*gag

Work in Progress: 201809

Invoicing sofware for service providers based on a Java EE multilayer architecture

(gl) Aplicación de facturación con arquitectura Java EE multicapa para empresas provedoras de servizos
(es) Aplicación de facturación con arquitectura Java EE multicapa para empresas proveedoras de servicios

  • Supervisión

    Emilio José Padrón González (UDC)

  • Descrición

    Se propone el desarrollo e implementación de una aplicación de facturación basado en Software Libre, con arquitectura Java EE multicapa, que pueda ser utilizado por cualquier proveedor de servicios con una cartera de clientes a los que se les facturen cuotas (importes fijos a facturar por la prestación del servicio) y consumos (importe derivado del uso de un determinado servicio) de forma periódica, pudiendo definirse distintos ciclos de facturación atendiendo a las necesidades del negocio.

    El sistema a desarrollar interactuará con otros sistemas externos, como puede ser el el gestor de clientes o el sistema de tarificación de consumos, a fin de mantener coherencia en los datos manejados por las distintas entidades del negocio. Entre las características que ofrecerá el sistema de facturación, además de la citada facturación, se encuentra la posibilidad de definir promociones, bien sean descuentos sobre el total de la factura o sobre un concepto o conjunto de conceptos en particular (cuotas o consumos), aplicar/eliminar cuotas, consumos y/o promociones, aplicar rectificaciones sobre excesos o defectos de facturación de ciclos pasados, consulta de datos de facturas para un determinado cliente o la extracción de informes a través de consultas personalizadas.

  • Obxectivos concretos

    El objetivo del TFG es realizar el análisis, diseño e implementación de una aplicación de facturación basado en Software Libre, con arquitectura Java EE multicapa, que genere las facturas correspondientes a los clientes de una empresa atendiendo a los elementos facturables que tengan asociados, tipo impositivo aplicable y a las características definidas para la facturación de los mismos.

    Dicho sistema permitirá:

    • Definir distintos ciclos de facturación a aplicar según período de facturación, tipología de clientes sobre los que aplicar, recurrencia de ejecución…
    • Comunicación con plataformas externas (interfaz de contratación, sistema de tarificación, plataformas de impresión… ) tanto para recabar información como para el mantenimiento congruente de datos.
    • Posibilidad de modificación de datos (altas/bajas/modificaciones) desde el propio facturador, así como la definición de cuotas/promociones atendiendo a conceptos como recurrencia, importe a facturar/descontar, período de vigencia, etc.
    • Consultas de datos facturados/pendientes de facturar para un determinado cliente.
    • Posibilidad de extracción de informes para usuarios a través de consultas personalizadas.
  • Metodoloxía a seguir

    Se utilizará una metodología basada en técnicas de desarrollo iterativo o incremental

  • Fases a desenvolver

    Análisis de requisitos: identificación de las necesidades a cubrir por el sistema a desarrollar.

    Diseño: desarrollo de las especificaciones para el producto a crear.

    Desarrollo: codificación del software a desarrollar.

    Pruebas: realización de pruebas para validar que se cumplen los requisitos especificados.

  • Materiais e/ou medios necesarios

    • JSE JDK y JEE SDK
    • Entorno de desarrollo integrado JEE (IDE) open source
    • Base de datos relacional open source
    • Herramientas de modelado open source
    • Elementos necesarios para el desarrollo de web services: frameworks, contenedor web, etc.

Work in Progress: 20200319

(gl) Análise de datos en liña para simulacións científicas con Apache Flink
(es) Análisis de datos en línea para simulaciones científicas con Apache Flink

  • Supervision

    Bruno Raffin (Inria Rhône-Alpes, Univ. Grenoble Alpes)
    Emilio José Padrón González (UDC)

  • Brief description

    Large-scale simulations are producing an ever-growing amount of data that is becoming prohibitively costly, in terms of time and energy, to save to disks, and next to retrieve and process during the post-hoc data analysis phase. To circumvent this bottleneck, in-situ analytics [1] proposes to start processing data online, as soon as made available by the simulation in the memories of the compute nodes (or using other nodes in the same cluster, known as in-transit analysis). The benefits are:

    • Raw data produced by the simulation can start to be reduced before moving out of the compute nodes, saving on data movements and on the amount of data to store to disk.
    • Part of data analysis can be performed on the same supercomputer as the one booked for the simulation. The process can be massively parallelized, reading data from memory and not from disk, reducing the time for performing these tasks.

    This integration of data analytics with large-scale simulations represents a new kind of workflow. Scientists need to rethink the way to use the available data movement and storage budgets and the way to take advantage of the compute resources for advanced data processing. So far, only a few framework prototypes have been developed to investigate some key concepts, with experiments with simple analysis scenarios.

    The goal of this project proposal is to investigate and develop algorithms to enable advanced in-situ/in-transit processing of scientific data from numerical simulations with the ‘Big Data’ framework Apache Flink. Map/Reduce solutions where first targeting batch data processing. But needs for processing continuou streams of data like tweets led to a new breed of tools like Flink [2] able to connect to stream sources and trigger on-line analysis every time a user defined window of events being filled. These stream processing approaches have only recently been investigated for analysing results from large scale parallel simulations [3].

    But in-situ processing can be seen as special case of stream processing where the data are produced not by a web server, but by a large scale parallel simulation. Expected benefits include a user interface that does not require extensive parallel expertise to develop analysis kernels, kernels that can be used for both in-situ an post-hoc analysis, interoperability with advanced massive key/value stores such as Cassandra, out-of-the-box support for fault tolerance or multi-tenant analysis execution.

    [1] Lessons Learned from Building In Situ Coupling Frameworks.
    Matthieu Dorier, Matthieu Dreher, Tom Peterka, Gabriel Antoniu, Bruno Raffin, Justin M. Wozniak.
    ISAV 2015 – First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (held in conjunction with SC15), Nov 2015, Austin, United States.
    https://hal.inria.fr/hal-01224846
    [2] Apache Flink: Scalable Stream and Batch Data Processing.
    https://flink.apache.org
    [3] In-Transit Molecular Dynamics Analysis with Apache Flink.
    Henrique C. Zanuz, Bruno Raffin, Omar A. Mures, Emilio J. Padrón.
    ISAV 2018 – Fourth Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (held in conjunction with SC18), Nov 2018, Dallas, United States.
    https://hal.inria.fr/hal-01889939

  • Specific objectives

    • The main objective of this project is to develop analysis kernels for the online processing of scientic data from large-scale numerical simulations with Flink.

    • These kernels will operate within a current work-in-progress HPC infrastructure for in-transit analysis of scientific data based on Flink.

    • The specific applicative domain(s) will be determined during the development of the project, but we will probably target (at least) Molecular Dynamics simulations.

  • Methodology

    An Agile development method will guide the project, with relatively short sprints to build the different analysis kernels, after a preliminary work of study and documentation.

  • Development steps

    • Analysis of requirements and project scheduling, according to student disponibility.

    • Study and documentation.

      • The Map/Reduce paradigm and the framework Apache Flink.
      • Molecular Dynamics simulations (and other numerical simulations we can target to write online analysis kernels).
    • Incremental, iterative work sequences (sprints) to develop analysis kernels using Flink and integrate them in the existing work-in-progress HPC infrastructure for in-transit analysis of scientific data based on Apache Flink.

  • Material

    • Personal computer with internet access.

    • Access to HPC resources will be provided to the student.

Work in Progress: 20200626

Deep Learning Approach for Point-based Rendering

(gl) Unha aproximación Deep Learning para render de nubes de puntos
(es) Una aproximación Deep Learning para render de nubes de puntos

  • Breve descrición

    Unha nube de puntos é un conxunto de datos asociados a posicións puntuais no espazo. As nubes de puntos xeralmente son producidas por escáneres 3D, que miden gran cantidade de puntos nas superficies externas dos obxectos que os rodean, xerando conxuntos de datos que poden conter unha alta densidade de puntos.

    As nubes de puntos utilízanse para moitos propósitos, como para crear modelos CAD en 3D, para pezas manufacturadas, para metroloxía e inspección de calidade, e para multitude de aplicacións de visualización, animación e renderizado.

    Para unha visualización de calidade dunha nube de puntos de densidade arbitraria precísase de técnicas de renderizado específicas para o traballo con este tipo de primitivas gráficas, xa que tanto o hardware das tarxetas gráficas como o pipeline de render que expoñen as principais APIS gráficas están deseñados para a visualización de polígonos (triángulos, nomeadamente). Estas técnicas coñécense polo nome de Renderizado Baseado en Puntos (Point-based Rendering).

    O punto é unha primitiva adimensional, e unha nube de puntos non ten topoloxía, o que dificulta a obtención de vectores normais que poidan axudar na obtención dunha representación visual que permita reconstruír superficies sen buratos (para o que se adoita empregar unha técnica denominada «splatting»). O feito de que as nubes de puntos procedentes dunha captura adoiten ter ademais bastante ruído tampouco axuda no proceso de estimación de normais, que non é para nada trivial.

    As arquitecturas de Deep learning así como deep neural networks, deep belief networks e recurrent neural networks, estanse a aplicar con éxito en campos como visión artificial, recoñecemento de voz, procesamiento de linguaxe natural, recoñecemento de audio, filtrado de redes sociais, tradución entre máquinas, bioinformática, etc. Nalgunha destas disciplinas estas arquitecturas están a obter resultados comparables ou nalgúns casos superiores aos expertos humanos.

    O primeiro obxectivo deste proxecto é a implementación do estado da arte en técnicas Point-based Rendering e en algoritmos de estimación de normais para unha visualización de calidade de nubes de puntos de densidade arbitraria. Un segundo obxectivo é a implementación dun método para a estimación de normais empregando técnicas de Deep Learning, tentando mellorar os resultados obtidos mediante a aplicación de técnicas tradicionais.

    Para as tarefas de visualización farase uso da API gráfica multiplataforma OpenGL, mentres que para a parte de Deep Learning empregaremos TensorFlow e Keras.

  • Objetivos concretos

    • Desenvolver un visualizador para nubes de puntos en OpenGL que implemente as técnicas do estado da arte en renderizado de puntos, incluíndo unha estimación de normais baseada nos habituais métodos numéricos.

    • Propoñer e implementar un método de estimación de normais baseado nunha arquitectura Deep Learning que permita substituír (e estimamos que mellorar) ás técnicas habituais.

    • A aplicación desenvolvida será multiplataforma e permitirá avaliar todos os algoritmos implementados.

  • Metodoloxía

    Utilizaremos unha metodoloxía áxil, con sprints relativamente curtos para dar forma ás distintas tareas. Todo isto despois dun estudo e documentación da materia preliminar.

  • Fases principais do traballo

    1. Análisis de requisitos e planificación do proxecto.

    2. Estudo e documentación.

      a. Técnicas PRB e estimación de normais. b. C++, OpenGL, TensorFlow e Keras.

    3. Sprints incrementais para desenvolver técnicas de PBR para a visualización, e de estimacións de normais con métodos numéricos.

    4. Sprints incrementais para desenvolver unha aproximación para a estimación de normais con Deep Learning.

  • Material

    Un ordenador persoal con GPU e acceso a internet.

Work in Progress: 20200318

Implementation of an Automatic Camera Operator using Deep Reinforcement Learning

(gl) Implementación dun operador de cámara automático usando Deep Reinforcement Learning
(es) Implementación de un operador de cámara automático usando Deep Reinforcement Learning

  • Brief description

    Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. The problem, due to its generality, is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, statistics and genetic algorithms. In the operations research and control literature, reinforcement learning is called approximate dynamic programming, or neuro-dynamic programming [1] [2].

    The problems of interest in reinforcement learning have also been studied in the theory of optimal control, which is concerned mostly with the existence and characterization of optimal solutions, and algorithms for their exact computation, and less with learning or approximation, particularly in the absence of a mathematical model of the environment. Reinforcement algorithms that incorporate deep learning can beat world champions at the game of Go as well as human experts playing numerous Atari video games. Although that may sound trivial, it’s a vast improvement over their previous accomplishments, and the state of the art is progressing rapidly.

    A game engine is a software-development environment designed for people to build video games. Developers use game engines to construct games for consoles, mobile devices, and personal computers. The core functionality typically provided by a game engine includes a rendering engine (“renderer”) for 2D or 3D graphics, a physics engine or collision detection (and collision response), sound, scripting, animation, artificial intelligence, networking, streaming, memory management, threading, localization support, scene graph, and may include video support for cinematics. Implementers often economize on the process of game development by reusing/adapting, in large part, the same game engine to produce different games [3] or to aid in porting games to multiple platforms. Since agents need an accurate virtual representation of the task at hand, 3D engines come in handy to create computational environments in which we can train them in parallel. This makes exhausting our computational resources possible to accelerate the aforementioned training.

    In this project, we will leverage Deep Reinforcement Learning and game engines to model a typical sports scene and teach an agent to capture the action on camera. First, a scene which resembles a sports match will be created using the chosen game engine. Next, we will test out different Deep Reinforcement Learning algorithms (Discrete-Action DQN, Parametric-Action DQN, Double DQN, Dueling DQN, Dueling Double DQN, DDPG (DDPG), Soft Actor-Critic (SAC)) to see which one fits this problem best.

    The specific application domains would be broadcasting, games, surveillance, etc. The solutions developed in this project will be integrated in an open source game engine, probably Godot [4].

    [1] Reinforcement learning and markov decision processes.
    Martijn van Otterlo, Marco Wiering.
    In: Wiering M., van Otterlo M. (eds) Reinforcement Learning. Adaptation, Learning, and Optimization,
    vol 12. Springer, Berlin, Heidelberg. 2012.
    DOI: 10.1007/978-3-642-27645-3_1

    [2] Reinforcement Learning: A Survey.
    Leslie P. Kaelbling, Michael L. Littman, Andrew W. Moore.
    Journal of Artificial Intelligence Research 4, pp. 237-285. 1996.
    DOI: 10.1613/jair.301

    [3] 3D Game Engine Programming (Game Development Series).
    Stefan Zerbst, Oliver Düvel.
    Course Technology PTR; 1 edition (June 30, 2004)
    ISBN-10: 1592003516

    [4] Godot Game Engine.
    https://godotengine.org

  • Specific objectives

    • The main objective of this project is to develop the described environment in a game engine and train a RL agent to solve the aforementioned task.

    • The student will explore new innovative Deep Learning methods.

    • The Automatic Camera Operator implemented will be integrated in an open source game engine.

  • Methodology

    An Agile development method will guide the project, with relatively short sprints to build the different tasks, after a preliminary work of study and documentation.

  • Development steps

    • Analysis of requirements and project scheduling, according to student disponibility.

    • Study and documentation.

      • Game engines.
      • TensorFlow, Horizon.
    • Incremental, iterative work sequences (sprints) to develop a 3D environment that models an sport.

    • Incremental, iterative work sequences (sprints) to develop a Deep Reinforcement Learning approach for imitating a camera operator.

  • Material

    • Personal computer with GPU and internet access.

Work in Progress: 20200317

Deep Learning chroma keyer implementation

(gl) Implementación dun chroma keyer usando Deep Learning
(es) Implementación de un chroma keyer usando Deep Learning

  • Brief description

    Chroma key compositing, or chroma keying, is a visual effects/post-production technique for compositing (layering) two images or video streams together based on color hues (chroma range). The technique has been used heavily in many fields to remove a background from the subject of a photo or video – particularly the newscasting, motion picture, and video game industries. A color range in the foreground footage is made transparent, allowing separately filmed background footage or a static image to be inserted into the scene.

    The chroma keying technique is commonly used in video production and post-production. This technique is also referred to as color keying, colour-separation overlay (CSO; primarily by the BBC), or by various terms for specific color-related variants such as green screen, and blue screen – chroma keying can be done with backgrounds of any color that are uniform and distinct, but green and blue backgrounds are more commonly used because they differ most distinctly in hue from most human skin colors. No part of the subject being filmed or photographed may duplicate the color used as the backing [1].

    Deep learning is an aspect of artificial intelligence (AI) that is concerned with emulating the learning approach that human beings use to gain certain types of knowledge. At its simplest, deep learning can be thought of as a way to automate predictive analytics. While traditional machine learning algorithms are linear, deep learning algorithms are stacked in a hierarchy of increasing complexity and abstraction. Each algorithm in the hierarchy applies a nonlinear transformation on its input and uses what it learns to create a statistical model as output. Iterations continue until the output has reached an acceptable level of accuracy. The number of processing layers through which data must pass is what inspired the label deep [2].

    Because deep learning models process information in ways similar to the human brain, models can be applied to many tasks people do. Deep learning is currently used in most common image recognition tools, NLP processing and speech recognition software. These tools are starting to appear in applications as diverse as self-driving cars and language translation services.

    The goal of this project is to segment objects of interest from the background in real-time using SegNet. SegNet is a deep encoder-decoder architecture for multi-class pixelwise segmentation researched and developed by members of the Computer Vision and Robotics Group at the University of Cambridge, UK [3]. This segmentation will be used to provide a 3D chroma key in real time. Specific application domains would be games, broadcasting, etc.

    [1] The Green Screen Handbook: Real-World Production Techniques
    Jeff Foster.
    Sybex; 1 edition (March 15, 2010)
    ISBN-10: 0470521074

    [2] Deep Learning in Neural Networks: An Overview
    Juergen Schmidhuber.
    Neural Networks, Vol 61, pp 85-117, Jan 2015
    DOI: 10.1016/j.neunet.2014.09.003

    [3] SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling
    Vijay Badrinarayanan, Ankur Handa, Roberto Cipolla. 2015
    https://arxiv.org/abs/1505.07293

  • Specific objectives

    • The main objective of this project is to develop a 3D chroma keyer using a SegNet-based architecture.

    • The student will explore new innovative methods to perform chroma keying in real-time.

    • A standalone proof-of-concept application will be developed.

  • Methodology

    An Agile development method will guide the project, with relatively short sprints to build the different tasks, after a preliminary work of study and documentation.

  • Development steps

    • Analysis of requirements and project scheduling, according to student disponibility.

    • Study and documentation.

      • Chroma keying.
      • TensorFlow, C++, libav.
      • SegNet.
    • Incremental, iterative work sequences (sprints) to develop a real-time chroma keyer.

  • Material

    • Personal computer with GPU and internet access.

Work in Progress: 20200317 :Multimedia:Video:Denoising:Super-resolution:Deep__Learning:AI:

Sports video super-resolution and denoising using Deep Learning

(gl) Super-resolución e eliminación de ruído en vídeos deportivos empregando Deep Learning
(es) Super-resolución y eliminación de ruido en vídeos deportivos usando Deep Learning

  • Brief description

    Super-resolution imaging is the process of enhancing the detail of an image. A low resolution image (LR) is taken as input and it will be upscaled to a higher resolution output (HR). Usually, the input image has a “lower resolution” due to a smaller spatial resolution (i.e. size), due to a result of degradation (such as blurring) or both. We can relate the HR and LR images through the following equation:

    LR = degradation(HR)

    Clearly, on applying a degradation function, we obtain the LR image from the HR image. But, can we do the inverse? Of course, we usually do not know the degradation function beforehand. Directly estimating the inverse degradation function is an ill-posed problem. In spite of this, Deep Learning (DL) techniques have proven to be effective for super-resolution, with promising results in current research.

    Most DL based super-resolution models are trained using Generative Adversarial Networks (GANs). A generative adversarial network (GAN) is a class of machine learning model designed by Ian Goodfellow and his colleagues in 2014 [1]. Two neural networks contest with each other in a game (in the form of a zero-sum game, where one agent’s gain is another agent’s loss).

    Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proven useful for semi-supervised learning, fully supervised learning, and reinforcement learning.

    In this project we will analyze the current state of the art in DL super-resolution and will experiment with some of the bleeding edge techniques, training a model for our custom dataset and, finally, extending that model to perform denoising as well (i.e. noise reduction/removal). This will allow further testing in a video streaming context, which requires low bandwidth video transmission.

    [1] Goodfellow, Ian; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua (2014). Generative Adversarial Networks.

  • Specific objectives

    • Research deep learning super-resolution techniques

    • Develop a deep learning model to achieve super-resolution + denoising of an input video stream.

    • The specific application domain(s) will be broadcasting, surveillance, etc.

  • Methodology

    An Agile development method will guide the project, with relatively short sprints to build the different tasks, after a preliminary work of study and documentation.

  • Development steps

    • Analysis of requirements and project scheduling, according to student disponibility.

    • Study and documentation.

      • GANs.
      • TecoGAN, DRLN…
      • TensorFlow, Pytorch.
    • Incremental, iterative work sequences (sprints) to develop and train the chosen model.

    • Incremental, iterative work sequences (sprints) to test the video streaming use case.

  • Material

    • Personal computer with GPU and internet access.