Richard Zanibbi, Assistant Professor, Department of Computer Science
Rochester Institute of Technology

Recent Developments in Math Recognition and Retrieval

Wednesday, March 14, 4:00 PM
Packard Lab 466

 Reception Prior to talk at 3:30 PM in Packard Lobby

Abstract: A team in the Document and Pattern Recognition Lab at RIT is working on new methods for retrieving math in documents, using mathematical expressions as queries. Our goal is to develop retrieval tools that are intuitive to use, both for experts and (perhaps more importantly) non-experts. An overview of recent and ongoing research in our lab will be presented, including techniques for image-based retrieval of expressions in document images using handwritten and typeset expression images, retrieval using LaTeX strings, and 'min,' a web-based system for recognizing handwritten mathematical expressions. Our system for retrieving handwritten queries in document images worked surprisingly well in our first human experiment, on average locating almost half (43.3%) of the query in its original location in the top 10 matches (10 users, writing 20 test expressions taken from 200 clean document images). Our two LaTeX-based retrieval systems had similar performance in their first comparison (10 users, 10 queries, 50 LaTeX documents taken from the arXiv), with our Substitution Tree-based system having higher average top-1 precision (100% vs. 89%), and our keyword-based system using Term Frequency - Inverse Document Frequency weighting (TF-IDF) having slightly higher average top-5 precision (51% vs. 48%). I will also discuss the Hidden Markov Model (HMM) used for symbol recognition in the 'min' system, which had an 82.9% recognition rate and a top-5 recognition rate of 97.8% for 93 symbol classes across 20 writers in a recent experiment. I will close the talk by identifying opportunities for future work, including the possibility of adapting techniques for math retrieval to other non-textual document elements such as chemical diagrams, tables, and figures.
*Joint work with Lei Hu, Li Yu, Thomas Schellenberg, Bo Yuan, Richard Pospesel, Kevin Hart, and David Stalnaker.

Bio: Richard Zanibbi is an Assistant Professor in the Department of Computer Science at the Rochester Institute of Technology (Rochester, NY), where he directs the Document and Pattern Recognition Lab. His research group works on algorithms and tools for pattern recognition systems, with an emphasis on applications for typeset, handwritten, and electronic documents. He holds a PhD in Computer Science from Queen's University in Kingston, Canada, and was an NSERC postdoctoral fellow at the Centre for Pattern Recognition and Machine Intelligence (CENPARMI) in Montreal before joining RIT. Dr. Zanibbi was one of the main contributors to the Freehand Formula Entry System (FFES), an influential pen-based equation editing prototype, and is the conference Co-Chair for the SPIE Document Recognition and Retrieval XIX (2012) and XX (2013) conferences. He recently published a survey of math recognition and retrieval research, to appear in a forthcoming issue of the International Journal on Document Analysis and Recognition.

© 2014-2016 Computer Science and Engineering, P.C. Rossin College of Engineering & Applied Science, Lehigh University, Bethlehem PA 18015.