Kenny Davila @ kdavila.com

Lecture Videos

Chart Recognition

Math Information Retrieval

Math Symbol Recognizer

LectureMath: Extracting and Analyzing Lecture Video Content

This is a continuation of the AccessMath project (see below). This broadens the research scope to work with a wider set of lecture video recordings.

FCN-LectureNet

Code and data from this project can be found Here.

AccessMath: Whiteboard Content Extraction and Retrieval from Lecture Videos

AccessMath is a project originally conceived with the goal of helping students with low vision both during lectures and outside of the classroom. However, the tools produced by this project will be helpful for other students in general.

A collection of lecture videos contains a large amount of information which is hard to retrieve without the proper labels or tags. In AccessMath, we are worked on methods for extraction and indexing of the content of math lecture videos with minimal human intervention. After indexing, AccessMath provides methods for retrieval based on the visual similarity between the provided query images and the whiteboard content stored in the index.

Source Video Frame	Content Extracted

There are different challenges involved in the process of whiteboard content extraction from the videos. After extraction, the content is indexed for visual-based retrieval.

Binarization	Background Identification	Background Removal	CC Stability Analysis

Speaker Action Analysis

During the lecture, the speaker performs several actions which change the handwritten content on the whiteboard. We have developed methods for speaker action classification based on pose estimations.

Erasing	Writing

Source Code: The source code for this project can be found here

Lecture Video Navigation

The automated extraction of unique handwritten content from the whiteboard enables more advanced applications such as content navigation based on key-frames. A demo of the system can be seen here. Our friends from Math2me have enabled us to create a demo of the navigation system using some videos from their channel. This other demo can be seen here.

Tangent-V

We developed the Tangent-V system originally for indexing and retrieval of the whiteboard content extracted from lecture videos. The system builds representations of the handwritten content using Line-Of-Sight graphs.

Related Publications

K. Davila, F. Xu, J. Molina, S. Setlur, V. Govindaraju. "Synthetic Data Generation for Semantic Segmentation of Lecture Videos". International Conference on Frontiers in Handwriting Recognition (ICFHR 2022). Springer.

K. Davila, F. Xu, S. Setlur, V. Govindaraju. "FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos". IEEE Access (Access). Volume 9. (Jul. 2021). IEEE.

B.U. Kota, A. Stone, K. Davila, S. Setlur, V. Govindaraju. "Automated Whiteboard Lecture Video Summarization by Content Region Detection and Representation". International Conference on Pattern Recognition (ICPR 2020). IEEE.

F. Xu, K. Davila, S. Setlur, V. Govindaraju. "Skeleton-based methods for speaker action classification on lecture videos". International Conference on Pattern Recognition Workshops (ICPR-W 2020). Springer.

F. Xu, K. Davila, S. Setlur, V. Govindaraju. "Content Extraction from Lecture Video via Speaker Action Classification Based on Pose Information". International Conference on Document Analysis and Recognition (ICDAR 2019). (BEST STUDENT PAPER AWARD)

B.U. Kota, S. Ahmed, A. Stone, K. Davila, S. Setlur, V. Govindaraju. "Summarizing Lecture Videos by Key Handwritten Content Regions". Camera Based Document Analysis and Recognition (CBDAR 2019). (BEST PAPER AWARD)

B.U. Kota, K. Davila, A. Stone, S. Setlur, V. Govindaraju. "Generalized framework for summarization of fixed-camera lecture videos by detecting and binarizing handwritten content". International Journal on Document Analysis and Recognition (IJDAR), Volume 22, Issue 3. (Sept. 2019).

K. Davila, R. Zanibbi. "Visual Search Engine for Handwritten and Typeset Math in Lecture Videos and LATEX Notes". In Proc. International Conference on Frontiers in Handwriting Recognition (ICFHR 2018). IEEE. (BEST PAPER AWARD)

B.U. Kota, K. Davila, A. Stone, S. Setlur, V. Govindaraju. "Automated Detection of Handwritten Whiteboard Content in Lecture Videos for Summarization". In Proc. International Conference on Frontiers in Handwriting Recognition (ICFHR 2018). IEEE.

K. Davila, R. Zanibbi. "Whiteboard Video Summarization via Spatio-Temporal Conflict Minimization". In Proc. International Conference on Document Analysis and Recognition (ICDAR 2017). IEEE.

K. Davila. "Appearance-Based Retrieval of Mathematical Notation in Documents and Lecture Videos". In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR 2016). ACM.

K. Davila, A. Agarwal, R. Gaborski, R. Zanibbi and S. Ludi. "AccessMath: Indexing and Retrieving Video Segments Containing Math Expressions Based on Visual Similarity". In IEEE Western New York Image Processing Workshop (WNYIPW 2013). IEEE.

Technical Report: MS Project Report (Updated: May 9, 2013)

Competition on HArvesting Raw Tables from Information Graphics (CHART-Infographics)

Recent years have seen an increased interest in automated recognition of statistical graphics (charts). However, as it can be read in our extensive literature review, there have been limited comparisons between existing methods due to the lack of standarized benchmarks. For this reason, we have proposed and executed the CHART-Infographics competitions. Further information about the competition and the corresponding tools and data produced can be found in the official competition website.

CHART-Infographics Tasks

Related Publications

K. Davila, F. Xu, S. Ahmed, D.A. Mendoza, S. Setlur, V. Govindaraju. "ICPR 2022: Challenge on Harvesting Raw Tables from Infographics (CHART-Infographics)". International Conference on Pattern Recognition (ICPR 2022). IEEE

K. Davila, C. Tensmeyer, S. Shekhar, H. Singh, S. Setlur, V. Govindaraju. "ICPR 2020-competition on harvesting raw tables from infographics". International Conference on Pattern Recognition Workshops (ICPR-W 2020). Springer.

K. Davila, S. Setlur, D. Doermann, B.U. Kota, V. Govindaraju. "Chart Mining: A Survey of Methods for Automated Chart Analysis". IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Early Access.

K. Davila, B.U. Kota, S. Setlur, V. Govindaraju, C. Tensmeyer, S. Shekhar, R. Chaudhry. "ICDAR 2019 Competition on Harvesting Raw Tables from Infographics (CHART-Infographics)". International Conference on Document Analysis and Recognition (ICDAR 2019).

Tangent: Math Search Engine

Tangent is a scalable math search engine originally developed at DPRL.

Tangent-3

Tangent version 3 has been developed in collaboration with Frank W. Tompa and Andrew Kane from University of Waterloo. This version of the search engine uses a two-stage retrieval method. On the first stage, a core-engine quickly finds good candidate matches from large databases using pairs of symbols from a Symbol Layout Tree (SLT) used to represent each formula. On the second stage, a re-ranker applies a finer matching method which is able to unify variables and has partial support for wildcard expansion. My main contributions to this project are the re-ranking functionality and visualization tools.

Matching a query with wildcard and variable unification	SLT Representation of a candidate match

Tangent-S

The latest version, Tangent-S, is now capable of indexing mathematical expressions using semantic operator trees (OPT). It also adds an optional third layer to the retrieval system that uses linear regression to combine multiple similarity metrics for better ranking of formulas. Tangent-S was included as a baseline in the ARQMATH challenge in CLEF 2020, and the system produced the best results for ad-hoc formula retrieval.

Source Code and Data: Tangent @ DPRL website

Tangent-V: Graph-based Search Engine

Tangent-V is a visual search engine based on graphs. To use with mathematical expressions, we have employed Line-of-Sight graphs and we have done retrieval of mathematical expressions fully based on visual similarity. Unlike Tangent-S which directly works with symbolic representations, the Tangent-V system works with images both in PNG and PDF format. The system is domain agnostic except for the basic node labels which are known for PDFs and inferred for PNGs using a math symbol recognizer.

Related Publications

Y. Diaz, G. Nishizawa, B. Mansouri, K. Davila, R. Zanibbi. "The MathDeck Formula Editor: Interactive Formula Entry Combining LaTeX, Structure Editing, and Search". Extended Abstracts of Conference on Human Factors in Computing (CHI 2021). ACM.

S. Ahmed, K. Davila, S. Setlur, V. Govindaraju. "Equation Attention Relationship Network (EARN): A Geometric Deep Metric Framework for Learning Similar Math Expression Embedding". International Conference on Pattern Recognition (ICPR 2020). IEEE.

K. Davila, R. Joshi, S. Setlur, V. Govindaraju, R. Zanibbi. "Tangent-V: Math Formula Image Search Using Line-of-Sight Graphs". In Proc. European Conference on Information Retrieval (ECIR 2019).

K. Davila, R. Zanibbi. "Layout and Semantics: Combining Representations for Mathematical Formula Search". In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR 2017). ACM.

K. Davila, R. Zanibbi, A. Kane, FW Tompa. "Tangent-3 at the NTCIR-12 MathIR Task". Proceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies (NTCIR-12). 2016.

R. Zanibbi, A. Aizawa, M. Kohlhase, I. Ounis, G. Topic, K. Davila. "NTCIR-12 MathIR Task Overview.". Proceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies (NTCIR-12). 2016.

K. Davila. "Appearance-Based Retrieval of Mathematical Notation in Documents and Lecture Videos". In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR 2016). ACM.

R. Zanibbi, K. Davila, A. Kane, F.W. Tompa. "Multi-Stage Math Formula Search: Using Appearance-Based Similarity Metrics at Scale". In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR 2016) . ACM.

Math Symbol Recognizer

Giving the trace data of a math symbol, the math symbol recognizer can identify which math symbol it is. The main hypothesis is that math symbols can be recognized based mainly on the shape of their traces without considering how the shape was drawn. For this reason, the project uses adaptations of off-line features that mainly describe shape instead of the drawing process like other existing methods. Training and testing data came from the Competition on Recognition of On-line Handwritten Math Expressions (CROHME). Additional tests were performed using synthetic data generation which improved per-class recognition rates.

Examples of traces of handwritten math symbols supported by Math Symbol Recognizer

Off-line Features used for Math Symbol Recognition
Crossing Features	Fuzzy 2D Histogram	Fuzzy Histograms of Orientations

Related Publications

M. Mahdavi, M. Condon, K. Davila, R. Zanibbi. "LPGA: Line-of-sight parsing with graph-based attention for math formula recognition". International Conference on Document Analysis and Recognition (ICDAR 2019).

K. Davila, S. Ludi and R. Zanibbi. "Using Off-line Features and Synthetic Data for On-line Handwritten Math Symbol Recognition". 14th International Conference on Frontiers in Handwriting Recognition (ICFHR 2014). IEEE.

Source Code: Math Symbol Recognizer at GitHub

Computer Graphics

Artificial Intelligence

Stage Kreator 0.1 (2012)
Main menu	Editing terrain	Casting shadows	Full Scene

Game Demos
Xuul Kab Demo (2010)	Xuul Kab 3D Demo (2011)
Semi-Finalist submission for Imagine Cup 2010 Game Design competition. Hun Mixba'al (One-Zero) Team was: Wilfredo R. Guevara, Xenia Oliva, Carlos Sanchez and Kenny Davila.	Semi-Finalist submission for Imagine Cup 2011 Game Design competition. Hun Mixba'al (One-Zero) Team was: Wilfredo R. Guevara, Xenia Oliva, Mijail Bulgakov and Kenny Davila.

The Roll Dice Visualizer	Artificial Life: Surviving Robots
Using A* search algorithm to solve a simple puzzle. The dice starts at a given location and must reach a certain target location rolling. Two constraints are added here related to face with the number 1. The first one is that this face cannot be at the bottom at any time. The second one is that this face has to be on the top when the goal is reached.	Simulated life where robots are given an artificial brain represented by a Neural Network. To stay alive the robots must eat the green elements that appear on the simulated environment. Genetic Algorithms are applied to find weights for a simulated brain using neural networks. The longer a robot lives, the more fitness it has and the greater its changes of generating offspring.