Difference between revisions of "Sketch/Proposal"

From Eclipsepedia

Jump to: navigation, search
m
Line 22: Line 22:
 
== Architecture and Extensibility ==
 
== Architecture and Extensibility ==
  
This API is composed of three parts: a sketch '''Tool''', a '''Recognizer''' and a sketch '''Bank'''.  The figure below shows how the API integrates with GEF, through AbstractTool and GraphicalEditor.
+
This API is composed of three parts: a sketch '''Tool''', a '''Recognizer''' and a sketch '''Bank'''.  The figure below shows how the API integrates with GEF, through AbstractTool and GraphicalEditor. The upper part of the figure is an example of integration -- it shows a GEF editor (ShapesDiagramEditor) and three tools (for Square, Triangle and a connection). The integration is made through the SketchTool -- a tool that observes the user's clicks and drags and provides a visual feedback of what is being drawed on the editor. Once the sketch is finished, the tool also passes the points to the SketchManager, which in turn processes those points and updates the editor.
  
 
[[Image:Sketch_Class_Diagram.png |center|600px| Class diagram for Sketch API‎ ]]
 
[[Image:Sketch_Class_Diagram.png |center|600px| Class diagram for Sketch API‎ ]]
Line 30: Line 30:
 
[[Image:Levenshtein.png‎ |left|300px| Levenshtein applied to sketch Recognition‎ ]]
 
[[Image:Levenshtein.png‎ |left|300px| Levenshtein applied to sketch Recognition‎ ]]
  
The recognition is made using a chain of algorithms to take part of a decision. The SketchManager passes to SketchElementRecognizer the user's sketch as a list of points, and periodically probes it for a result. The SketchElementRecognizer creates the chain using a set of algorithms. If the first algorithm of the chain is not 'sure' about what the sketch means, it can pass the decision to the next one on the chain, returning an element or null, if no element could be determined.  
+
The recognition is made using a chain of algorithms to take part in a decision. The SketchManager passes to SketchElementRecognizer the user's sketch as a list of points, and periodically probes it for a result. The SketchElementRecognizer creates the chain using a set of algorithms. If the first algorithm of the chain is not 'sure' about what the sketch means, it can pass the decision to the next one on the chain, returning an element or null, if no element could be determined.  
  
 
The first algorithm on the current chain was made using an approach based on Levenshtein's algorithm for string distance. It receives the user's sketch transformed in a set of points and places them on a grid, then each point is transformed to a number according to the next point's position. This approach is described in detail here (http://www.springerlink.com/content/u0r7261w01248q1t/).  
 
The first algorithm on the current chain was made using an approach based on Levenshtein's algorithm for string distance. It receives the user's sketch transformed in a set of points and places them on a grid, then each point is transformed to a number according to the next point's position. This approach is described in detail here (http://www.springerlink.com/content/u0r7261w01248q1t/).  

Revision as of 17:52, 1 February 2010

Contents

Draft proposal

This is a draft proposal, feel free to contribute to it!

Introduction

Sketch provides an API to allow users to insert elements (EditParts) on GEF/GMF editors by capturing its gestures, typically using a pen on a tablet. The idea is to allow user might freely draw the representation of an element on the editor's area. The API would then transform this rough user's representation on something processable, interpreting what element the user meant to insert.

The API might learn from user's input, recognizing each sketch in the way it is drawn by different users. In cases when it is impossible to determine what element is meant by a sketch, the API 'asks' the user, through a small dialog, what he/she meant by that sketch. Thus the API is able to learn for the next times.

Sketch is proposed as an open source project under the Eclipse Technology Project (http://www.eclipse.org/technology/). This proposal is still in the Project Proposal Phase, and is being made in order to call for more community participation. You are invited to comment on and/or join the project. Please send all feedback to the http://www.eclipse.org/newsportal/thread.php?group=eclipse.technology.sketch newsgroup.

Background and Goal

Application prototype for Sketch API‎

With the increasing popularity of touch-enabled devices nowadays, it is getting more common to insert information directly on the screen, using the fingers or a pen, instead of using a mouse. This project aims to enrich user's interaction with GEF/GMF editors using the capabilities provided by touch/tablet devices.

The goal of this project is not only to allow users to insert elements using sketches, but also to represent elements on the palette using the users' own drawing.

The Sketch API is targeted primarily at editors with a large set of elements, in which the user have to search through the whole GEF palette in order to add them to the editor.


Architecture and Extensibility

This API is composed of three parts: a sketch Tool, a Recognizer and a sketch Bank. The figure below shows how the API integrates with GEF, through AbstractTool and GraphicalEditor. The upper part of the figure is an example of integration -- it shows a GEF editor (ShapesDiagramEditor) and three tools (for Square, Triangle and a connection). The integration is made through the SketchTool -- a tool that observes the user's clicks and drags and provides a visual feedback of what is being drawed on the editor. Once the sketch is finished, the tool also passes the points to the SketchManager, which in turn processes those points and updates the editor.

Class diagram for Sketch API‎

Sketch Recognition

Levenshtein applied to sketch Recognition‎

The recognition is made using a chain of algorithms to take part in a decision. The SketchManager passes to SketchElementRecognizer the user's sketch as a list of points, and periodically probes it for a result. The SketchElementRecognizer creates the chain using a set of algorithms. If the first algorithm of the chain is not 'sure' about what the sketch means, it can pass the decision to the next one on the chain, returning an element or null, if no element could be determined.

The first algorithm on the current chain was made using an approach based on Levenshtein's algorithm for string distance. It receives the user's sketch transformed in a set of points and places them on a grid, then each point is transformed to a number according to the next point's position. This approach is described in detail here (http://www.springerlink.com/content/u0r7261w01248q1t/).

For each element there is a set of stored words that describes its many forms, like drawn by the user. The recognizer compares each new sketch with the words of each element, calculating the distance between them. If the distance is smaller for a square than for a triangle, for example, then the element is probably a square.

The Chain of Responsibility pattern is applied to allow other algorithms to take part on the decision regarding what element is meant by a given sketch. Currently if a sketch is not recognized by the LevenshteinHandler, it is passed to ConnectionHandler, which analyses if the user is trying to connect two elements.

Project Scope

This project is targeted to final users and might be integrated with GEF editors through a small set of configurations, in which is possible to specify, among other things, the elements that should be recognized.

Future Roadmap

  • Add more and more robust interpreters at the recognition chain
  • Support for GEF
  • Support for GMF through sketch models related to .gmfgraph (one group of primary sketches for each graphical node)

Concerns

  • Recognizer's degradation - Through many 'wrong' inputs it is possible to make the recognizer incapable to do correct assumptions regarding the sketches. The user must be able to correct this behavior, rolling back to a correct state.


Packaging and Deployment

Sketch will ship in the form of an Eclipse update site...

Organization

Mentors

  • Chris Aniszczyk
  •  ??

Initial committers

  • Ugo Sangiorgi (ugo.sangiorgi at gmail.com)

Interested parties

  • Chris Aniszczyk
  • Mariot Chauvin