|
|
Current Projects
|
Mathematics
|
|
The goal of the MathPaper project is to enrich handwritten mathematics by blending extended mathematical notations and diagrams together and by focusing on pen-centric manipulations. Ultimately, MathPaper would provide students, teachers, and professionals with a fluid integration of standard symbolic mathematics with geometry, algorithms, and symbolic computational assistance, on devices ranging from PDAs to whiteboards. Toward this goal, we present a writer-independent math recognition engine that explores techniques for visualizing and correcting errors as well as rudimentary support for the larger vision.
Basic Math
MathPaper provides a virtual sheet of paper that recognizes the free-form handwritten entry of multiple mathematical expressions to provide symbolic and numerical computational assistance. It uses real-time recognition to drive a suite of interactive recognition feedback techniques, and supports gestural and widget-based interactive editing. MathPaper also explores novel approaches to entering and manipulating matrices, including those with elided elements.
The technology in our research prototype (see Resources, below) is being commercialized by Fluidity Software.
Algorithmic Extensions to Basic Math
The algorithmic extensions demonstrate a vision of how MathPaper can support handwritten entry of mathematical algorithms. The initial development shows how to incorporate flow of control and other programmatic notations in a pen-friendly way. In addition, it shows an early idea of how debugging can be treated as an annotative process.
Resources
MathPaper installer 1.1.0, for Tablet PCs. Remember this is a research prototype which is quite capable of crashing or giving wrong answers.
MathPaper installer 1.1.1. This version has support for pasting expressions into Word 2007 which are editable with Word's new math editor. To do this, copy as MathML (with namespace), and then just paste into Word 2007 (not earlier versions). I'm still keeping the copy as MathML (without namespace) option in there in case there's a program that doesn't handle the namespace.
MathPaper introduction video
Sketching matrices video
Publications
Chuanjun Li, Robert Zeleznik, Timothy Miller, and Joseph J. LaViola Jr.
Online Recognition of Handwritten Mathematical Expressions with Support for Matrices.
To appear in Proceedings of the 19th International Conference on Pattern Recognition. December 2008.
Robert Zeleznik, Timothy Miller, Chuanjun Li, and Joseph J. LaViola Jr.
MathPaper: Mathematical Sketching with Fluid Support for Online Computation.
In Smart Graphics: 9th International Symposium, SG 2008. Rennes, France, August 2008. Lecture Notes in Computer Science 5166, p. 20–32. Springer-Verlag: Berlin, Heidelberg. Winner of the Best Paper award for the conference, based on the paper, presentation, and demo.
Chuanjun Li, Timothy S. Miller, Robert C. Zeleznik, and Joseph J. LaViola Jr.
AlgoSketch: Algorithm Sketching and Interactive Computation.
In EUROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling. June 2008.
Joseph J. LaViola Jr., Anamary Leal, Timothy S. Miller, and Robert C. Zeleznik.
Evaluation of Techniques for Visualizing Mathematical Expression
Recognition Results.
In Graphics Interface, p. 131–138,
Canadian Human-Computer Communications Society, May 2008.
Robert Zeleznik, Timothy Miller, and Chuanjun Li.
Designing UI Techniques for Handwritten Mathematics.
In EUROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling, p. 91–98, August 2007.
|
|
ChemPad project
|
|
|
Molecules are inherently three-dimensional structures that are represented on paper and in books as two-dimensional figures of lines, letters and symbols. To succeed at organic chemistry in college or high school courses, students must develop the ability to visualize two-dimensional drawings as dynamic, three-dimensional structures.
ChemPad is a Tablet PC application with a pedagogical focus that generates 3D molecular structures from hand drawn digital ink rather than from traditional molecule construction interfaces. Molecules are inherently three-dimensional (3D) objects that are represented by chemists on paper and classroom blackboards by a system of two-dimensional (2D) notations.
ChemPad allows chemists to sketch molecules in a quick and natural fashion to generate 3D models. This capability gives ChemPad pedagogical value to student chemists who have had difficulty with 3D chemistry thinking and used ChemPad to overcome that hurdle.
Resources
2006 Chempad video
Latest ChemPad Video (2008 3D Model Algorithm)
Dana Tennison's ChemPad Ph.D. project website: http://graphics.cs.brown.edu/research/chempad/home.html
|
Lineogrammer project
|
|
|
|
|
The Lineogrammer project objective is to make the process of producing diagrams much easier and more efficient. The observation that most diagrams can be sketched with a pen in a few seconds, while using other interfaces (e.g., keyboard and mouse) would require much more time and seem tedious by comparison, is why we believe a pen-based interface is probably best for producing the kinds of diagrams most people need. However, an initial sketch can be many steps away from a completed diagram; what is needed is an end-to-end process that preserves the efficiency and ease with which an initial sketch can be made, yet produces appropriately precise drawings.
Lineogrammer demonstrates the utility of a perceptually-inspired approach to entering diagrams that provides the user with immediate access to editing techniques at two levels of abstraction: primitives and their constituent lines. Simple recognition techniques are used to controllably distinguish text strokes from shape strokes. The project provides a blend of gestural and widget-based UI techniques for precise diagram control including alignment, symmetry, and linear distribution.
Resources
Lineogrammer video - wmv version
Lineogrammer video - mov version
|
|
We envision an aggregation of applications that use post-WIMP, pen-centric interfaces to make computational assistance more natural and efficient; among such applications are MathPad, the Music Notepad, and ChemPad. We call that aggregation *Pad. The goal of the starPad SDK project is to make it easier for people to write these sorts of applications by providing a convenient interface for a broad layer of pen-centric functionality in addition to some research functionality. Currently, it is written for MS Windows .Net and uses WPF.
|
|
MathPad2
|
|
|
MathPad2 is a prototype Tablet PC application for mathematical problem solving. At the core of its functionality is the novel concept of mathematical sketching, making dynamic illustrations by combining handwritten mathematics and free-form diagrams. MathPad2 is designed so a user can create simple illustrations as if they were working with pencil and paper. Teachers can use MathPad2 to quickly create illustrations to be used in their lessons and students can use the application to aid in their studies.
Resources
Joseph LaViola's MathPad2 Ph.D. project website: http://www.cs.brown.edu/~jjl/mathpad/
MathPad^2 video (Full video) |
|
Fluid Inking is a generally applicable approach to augmenting the fluid medium of free-form inking with gestural commands. Our approach is characterized by four design criteria, including:
1) pen-based hardware impartiality: all interactions can be performed with a button-free stylus, the minimal input hardware requirement for inking, and the least common denominator device for pen-based systems ranging from PDAs to whiteboards;
2) performability: gestures use short sequences of simple and familiar inking interactions that require minimal targeting;
3) extensibility: gestures are a regular pattern of optional shortcuts for commands in an arbitrarily scalable menu system; and
4) discoverability: gesture shortcuts (analogous to modifier keys) are displayed in the interactive menu and are suggested with dynamic feedback during inking.
Resources
Publications: Robert C. Zeleznik and Timothy S. Miller. "Fluid Inking: Augmenting the Medium of Free-Form Inking with Gestures."
|
|
Music Notepad
|
|
|
The Music Notepad was a system for entering common music notation based on 2D gestural input. The key feature of the system was the look-and-feel of the interface which approximates sketching music with paper and pencil. A probability-based interpreter integrates sequences of gestural input to perform the most common notation and editing operations. The paper presented at UIST '98 described the user's model of the system, the components of the high-level recognition system, and a discussion of the evolution of the system including user feedback.
Resources
Music Notepad video
Publications: Andrew S. Forsberg, Mark K. Dieterich, and Robert C. Zeleznik. "The
Music Notepad", Proceedings of UIST '98, ACM SIGGRAPH.
The Music Notepad was ported to the Tablet PC for Microsoft—source available.
|
|
ReMarkable Texts
|
|
|
ReMarkable Texts was a digital notebook, utilizing the Tablet PC, to be used by students and professionals for taking notes on lectures and collaborative projects. The goal of the system was to provide the ease of traditional notetaking on paper while utilizing the power of a computer.
|
|
SKETCH
|
|
|
Sketching is often a great way to communicate ideas with the implicit knowledge that the idea is in the formative stages: speed is high because one is using approximate visual images with simple tools (pencil and paper); there is no need for precision or specialized knowledge and, in fact, precision can confuse a viewer into thinking that an idea is more complete and immutable than it actually is; and finally is the ease of making low-level correction and revision. Most 3D computer modeling systems are good at generating arbitrary views of precise 3D models and support high-level editing and revision.
SKETCH combined the advantages of both hand-drawn-representation and precise computer modeling in order to create an environment for rapidly conceptualizing and editing approximate 3D scenes. To achieve this, SKETCH used simple non-photorealistic rendering and a purely gestural interface that was based on simplified line drawings of primitives and allowed all operations to be specified within the 3D world.
Resources
1996 Sketch video
1997 Two-handed input Sketch video
SKETCH project history website: http://graphics.cs.brown.edu/research/sketch/
Publications: Robert C. Zeleznik, Kenneth Herndon, and John F. Hughes. SKETCH: An interface for sketching 3D scenes in Proceedings of SIGGRAPH '96.
|
MSpcc webmaster
|
|
