
Current Projects

Mathematics

The goal of the MathPaper project is to enrich handwritten mathematics by blending extended mathematical notations and diagrams together and by focusing on pencentric manipulations. Ultimately, MathPaper would provide students, teachers, and professionals with a fluid integration of standard symbolic mathematics with geometry, algorithms, and symbolic computational assistance, on devices ranging from PDAs to whiteboards. Toward this goal, we present a writerindependent math recognition engine that explores techniques for visualizing and correcting errors as well as rudimentary support for the larger vision.
Basic Math
MathPaper provides a virtual sheet of paper that recognizes the freeform handwritten entry of multiple mathematical expressions to provide symbolic and numerical computational assistance. It uses realtime recognition to drive a suite of interactive recognition feedback techniques, and supports gestural and widgetbased interactive editing. MathPaper also explores novel approaches to entering and manipulating matrices, including those with elided elements.
The technology in our research prototype (see Resources, below) is being commercialized by Fluidity Software.
Algorithmic Extensions to Basic Math
The algorithmic extensions demonstrate a vision of how MathPaper can support handwritten entry of mathematical algorithms. The initial development shows how to incorporate flow of control and other programmatic notations in a penfriendly way. In addition, it shows an early idea of how debugging can be treated as an annotative process.
Resources
MathPaper installer 1.1.0, for Tablet PCs. Remember this is a research prototype which is quite capable of crashing or giving wrong answers.
MathPaper installer 1.1.1. This version has support for pasting expressions into Word 2007 which are editable with Word's new math editor. To do this, copy as MathML (with namespace), and then just paste into Word 2007 (not earlier versions). I'm still keeping the copy as MathML (without namespace) option in there in case there's a program that doesn't handle the namespace.
MathPaper introduction video
Sketching matrices video
Publications
Chuanjun Li, Robert Zeleznik, Timothy Miller, and Joseph J. LaViola Jr.
Online Recognition of Handwritten Mathematical Expressions with Support for Matrices.
To appear in Proceedings of the 19th International Conference on Pattern Recognition. December 2008.
Robert Zeleznik, Timothy Miller, Chuanjun Li, and Joseph J. LaViola Jr.
MathPaper: Mathematical Sketching with Fluid Support for Online Computation.
In Smart Graphics: 9th International Symposium, SG 2008. Rennes, France, August 2008. Lecture Notes in Computer Science 5166, p. 20–32. SpringerVerlag: Berlin, Heidelberg. Winner of the Best Paper award for the conference, based on the paper, presentation, and demo.
Chuanjun Li, Timothy S. Miller, Robert C. Zeleznik, and Joseph J. LaViola Jr.
AlgoSketch: Algorithm Sketching and Interactive Computation.
In EUROGRAPHICS Workshop on SketchBased Interfaces and Modeling. June 2008.
Joseph J. LaViola Jr., Anamary Leal, Timothy S. Miller, and Robert C. Zeleznik.
Evaluation of Techniques for Visualizing Mathematical Expression
Recognition Results.
In Graphics Interface, p. 131–138,
Canadian HumanComputer Communications Society, May 2008.
Robert Zeleznik, Timothy Miller, and Chuanjun Li.
Designing UI Techniques for Handwritten Mathematics.
In EUROGRAPHICS Workshop on SketchBased Interfaces and Modeling, p. 91–98, August 2007.

ChemPad project


Molecules are inherently threedimensional structures that are represented on paper and in books as twodimensional figures of lines, letters and symbols. To succeed at organic chemistry in college or high school courses, students must develop the ability to visualize twodimensional drawings as dynamic, threedimensional structures.
ChemPad is a Tablet PC application with a pedagogical focus that generates 3D molecular structures from hand drawn digital ink rather than from traditional molecule construction interfaces. Molecules are inherently threedimensional (3D) objects that are represented by chemists on paper and classroom blackboards by a system of twodimensional (2D) notations.
ChemPad allows chemists to sketch molecules in a quick and natural fashion to generate 3D models. This capability gives ChemPad pedagogical value to student chemists who have had difficulty with 3D chemistry thinking and used ChemPad to overcome that hurdle.
Resources
2006 Chempad video
Latest ChemPad Video (2008 3D Model Algorithm)
Dana Tennison's ChemPad Ph.D. project website: http://graphics.cs.brown.edu/research/chempad/home.html
Lineogrammer project



The Lineogrammer project objective is to make the process of producing diagrams much easier and more efficient. The observation that most diagrams can be sketched with a pen in a few seconds, while using other interfaces (e.g., keyboard and mouse) would require much more time and seem tedious by comparison, is why we believe a penbased interface is probably best for producing the kinds of diagrams most people need. However, an initial sketch can be many steps away from a completed diagram; what is needed is an endtoend process that preserves the efficiency and ease with which an initial sketch can be made, yet produces appropriately precise drawings.
Lineogrammer demonstrates the utility of a perceptuallyinspired approach to entering diagrams that provides the user with immediate access to editing techniques at two levels of abstraction: primitives and their constituent lines. Simple recognition techniques are used to controllably distinguish text strokes from shape strokes. The project provides a blend of gestural and widgetbased UI techniques for precise diagram control including alignment, symmetry, and linear distribution.
Resources
Lineogrammer video  wmv version
Lineogrammer video  mov version

We envision an aggregation of applications that use postWIMP, pencentric interfaces to make computational assistance more natural and efficient; among such applications are MathPad, the Music Notepad, and ChemPad. We call that aggregation *Pad. The goal of the starPad SDK project is to make it easier for people to write these sorts of applications by providing a convenient interface for a broad layer of pencentric functionality in addition to some research functionality. Currently, it is written for MS Windows .Net and uses WPF.

MathPad^{2}


MathPad^{2} is a prototype Tablet PC application for mathematical problem solving. At the core of its functionality is the novel concept of mathematical sketching, making dynamic illustrations by combining handwritten mathematics and freeform diagrams. MathPad2 is designed so a user can create simple illustrations as if they were working with pencil and paper. Teachers can use MathPad2 to quickly create illustrations to be used in their lessons and students can use the application to aid in their studies.
Resources
Joseph LaViola's MathPad^{2} Ph.D. project website: http://www.cs.brown.edu/~jjl/mathpad/
MathPad^2 video (Full video) 
Fluid Inking is a generally applicable approach to augmenting the fluid medium of freeform inking with gestural commands. Our approach is characterized by four design criteria, including:
1) penbased hardware impartiality: all interactions can be performed with a buttonfree stylus, the minimal input hardware requirement for inking, and the least common denominator device for penbased systems ranging from PDAs to whiteboards;
2) performability: gestures use short sequences of simple and familiar inking interactions that require minimal targeting;
3) extensibility: gestures are a regular pattern of optional shortcuts for commands in an arbitrarily scalable menu system; and
4) discoverability: gesture shortcuts (analogous to modifier keys) are displayed in the interactive menu and are suggested with dynamic feedback during inking.
Resources
Publications: Robert C. Zeleznik and Timothy S. Miller. "Fluid Inking: Augmenting the Medium of FreeForm Inking with Gestures."

Music Notepad


The Music Notepad was a system for entering common music notation based on 2D gestural input. The key feature of the system was the lookandfeel of the interface which approximates sketching music with paper and pencil. A probabilitybased interpreter integrates sequences of gestural input to perform the most common notation and editing operations. The paper presented at UIST '98 described the user's model of the system, the components of the highlevel recognition system, and a discussion of the evolution of the system including user feedback.
Resources
Music Notepad video
Publications: Andrew S. Forsberg, Mark K. Dieterich, and Robert C. Zeleznik. "The
Music Notepad", Proceedings of UIST '98, ACM SIGGRAPH.
The Music Notepad was ported to the Tablet PC for Microsoft—source available.

ReMarkable Texts


ReMarkable Texts was a digital notebook, utilizing the Tablet PC, to be used by students and professionals for taking notes on lectures and collaborative projects. The goal of the system was to provide the ease of traditional notetaking on paper while utilizing the power of a computer.

SKETCH


Sketching is often a great way to communicate ideas with the implicit knowledge that the idea is in the formative stages: speed is high because one is using approximate visual images with simple tools (pencil and paper); there is no need for precision or specialized knowledge and, in fact, precision can confuse a viewer into thinking that an idea is more complete and immutable than it actually is; and finally is the ease of making lowlevel correction and revision. Most 3D computer modeling systems are good at generating arbitrary views of precise 3D models and support highlevel editing and revision.
SKETCH combined the advantages of both handdrawnrepresentation and precise computer modeling in order to create an environment for rapidly conceptualizing and editing approximate 3D scenes. To achieve this, SKETCH used simple nonphotorealistic rendering and a purely gestural interface that was based on simplified line drawings of primitives and allowed all operations to be specified within the 3D world.
Resources
1996 Sketch video
1997 Twohanded input Sketch video
SKETCH project history website: http://graphics.cs.brown.edu/research/sketch/
Publications: Robert C. Zeleznik, Kenneth Herndon, and John F. Hughes. SKETCH: An interface for sketching 3D scenes in Proceedings of SIGGRAPH '96.




MSpcc webmaster

