Sketch/Proposal

Draft proposal

This is a draft proposal, feel free to contribute to it!

Introduction

Sketch provides an API to allow users to insert elements (EditParts) on GEF/GMF editors by capturing its gestures, typically using a pen on a tablet. The idea is to allow user might freely draw the representation of an element on the editor's area. The API would then transform this rough user's representation on something processable, interpreting what element the user meant to insert.

The API might learn from user's input, recognizing each sketch in the way it is drawn by different users. In cases when it is impossible to determine what element is meant by a sketch, the API 'asks' the user, through a small dialog, what he/she meant by that sketch. Thus the API is able to learn for the next times.

Sketch is proposed as an open source project under the Eclipse Technology Project (http://www.eclipse.org/technology/). This proposal is still in the Project Proposal Phase, and is being made in order to call for more community participation. You are invited to comment on and/or join the project. Please send all feedback to the http://www.eclipse.org/newsportal/thread.php?group=eclipse.technology.sketch newsgroup.

Background and Goal

Nowadays, with the increasing popularity of touch-enabled devices, it is getting more common to enable users to insert information directly on the screen, using their fingers or a pen, instead of using a mouse. This project aims to enrich user's interaction with GEF/GMF editors using the capabilities provided by touch/tablet devices.

Traditionally, GEF editors uses one tool for each element of the domain, thus the user must select a tool from the palette and click at the editor, adding a new element to the diagram. The goal of Sketch is to enable users to draw and connect the elements inside the editor's area, using their own gestures.

The Sketch API is targeted primarily at editors with a large set of elements, in which the user have to search through the whole GEF palette in order to add them to the editor.

A video showing the current state of the API is here: http://vimeo.com/2252782

Architecture and Extensibility

This API is currently composed of three parts: a sketch Tool, a Recognizer and a sketch Bank. The figure below shows how the API integrates with GEF, through AbstractTool and GraphicalEditor. The upper part of the figure is an example of integration -- it shows a GEF editor (ShapesDiagramEditor) and three tools (for Square, Triangle and a connection). The integration is made through the SketchTool -- a tool that observes the user's clicks and drags and provides a visual feedback of what is being drawn on the editor. Once the sketch is finished, the tool also passes the points to the SketchManager, which in turn processes those points and updates the editor.

Sketch Recognition

Basically, the recognition is made using a chain of algorithms to take part in a decision. The SketchManager gives a list of points captured by the SketchTool to SketchElementRecognizer, and then periodically probes the recognizer for a result. The SketchElementRecognizer creates the chain using a set of algorithms. If the first algorithm of the chain is not 'sure' about what the sketch means, it can pass the decision to the next one on the chain, returning an element or null, if no element could be determined.

The first algorithm on the current chain was made using an approach based on Levenshtein's algorithm for string distance. It receives the user's sketch transformed in a set of points and places them on a grid, then each point is transformed to a number according to the next point's position. This approach is described in detail here (http://www.springerlink.com/content/u0r7261w01248q1t/).

For each element there is a set of stored words that describes its many forms, like drawn by the user. The recognizer compares each new sketch with the words of each element, calculating the distance between them. If the distance is smaller for a square than for a triangle, for example, then the element is probably a square.

The Chain of Responsibility pattern is applied to allow other algorithms to take part on the decision regarding what element is meant by a given sketch. Currently if a sketch is not recognized by the LevenshteinHandler, it is passed to ConnectionHandler, which analyses if the user is trying to connect two elements.

Project Scope

The project will provide an API and an exemplary editor.The API is targeted to developers who wish to add sketching capabilities to GEF/GMF editors, doing so through a small set of configurations, in which is possible to specify, among other things, the elements that should be recognized. The exemplary editor will show how to use the API and expose the provided features to an end user.

Future Roadmap

• Add more and more robust interpreters at the recognition chain
• Support for GMF through sketch models related to .gmfgraph (one group of primary sketches for each graphical node)

Concerns

• Recognizer's degradation - Through many 'wrong' inputs it is possible to make the recognizer incapable to do correct assumptions regarding the sketches. The user must be able to correct this behavior, rolling back to a correct state.
  • What about having a view for the list of recognized artifacts explicitly available for the users? I'm not sure how you have the internal recognition mechanism set up, but this refinement capability could be an integral part of ad hoc graphical language creation process. --Milesparker.gmail.com 18:36, 18 April 2010 (UTC)

Packaging and Deployment

Sketch will ship in the form of an Eclipse update site...

Organization

Mentors

• Chris Aniszczyk
• Ed Merks

Initial committers

• Ugo Sangiorgi (ugo.sangiorgi at gmail.com) (Project lead)
• Mariot Chauvin, Obeo

Interested parties

• Chris Aniszczyk
• Ed Merks
• Obeo
• Aurélien Pupier & Mickael Istria (BonitaSoft)
• Oisín Hurley
• Sierra Wireless (David Sciamma)
• Jens von Pilgrim (GEF3D)
• Kristian Duske (GEF3D)
• Carsten Reckord, Andreas Scharf (Yatta Solutions)
• Eraldo R. Fernandes ([1])
• Miles Parker (AMP, Metascape)
• Luis Carlos Moreira da Costa Regional Communities Brazil