Jump to: navigation, search

Orion/Testing

Orion
Website
Download
Community
Mailing ListForumsMattermostIRC
Bugzilla
Open
Help Wanted
Bug Day
Contribute
Browse Source

This page talks about testing the features in Orion. Testing can be done locally (run the server on you local machine) or online on orionhub.org or orion.eclipse.org. Any test(s) that have to be run in a certain environment will mention so.

Test Passes

Test Page Template

A handy template for creating new test pass wiki pages can be found here.

To keep a consistent feel / test item listing across test passes, this template should be used to create new test pages.

Features to be Tested

Test Suites

The test suites are not a feature, but it is mandatory that every tester run all of the test suites before manual testing.

Consult this wiki page for specific instructions running the Node, Java and client test suites.

Accessibility

This section manually tests keyboard accessibility (it does not cover screen reader accessibility testing).

  • move your mouse out of reach and cover your trackpad (no cheating!)
  • now navigate to every page, menu, dialog, slideout panel, widget, etc, using only the keyboard (typically with tab or shift+tab)
  • make sure there's a clear focus indicator on every focusable element, and you never get lost or stuck
  • note: to tab or shift+tab out of editor, type ctrl+m first to change the editor's "tab insert mode"
  • make sure the tab order makes sense, and you're not jumping around or going backwards when trying to go forward
  • make sure activating and using specialized controls is intuitive and efficient (eg. type chars in filter, enter should activate)
  • for a list of Orion's keyboard shortcuts, type alt+shift+? (Win) or ctrl+shift+? (Mac)
  • test the keyboard shortcuts in the list
  • test a real world scenario (eg. clone a repo, create a branch, edit a file, commit and push)
  • Keyboard reminders:
    • activate buttons with space/enter; links with enter
    • activate menus/submenus with enter or down/right arrow, menu items with enter
    • arrows/home/end/esc should all work within menus, but tab/shift+tab should close menu and navigate out of menu bar
    • space to check/uncheck a checkbox/radio
    • arrows to move to/select other radios in a radio group; tab/shift+tab should navigate out of radio group
    • down arrow (or alt+down) to drop down a combobox list; up/down to move in list, enter to select, esc to close
    • all dialogs should put focus on an element in the dialog when they open
    • all dialogs should close with escape
    • you should not be able to navigate out of modal dialogs with tab/shift+tab
    • when a dialog closes, focus should go back to the trigger element
    • shift+F10 for context menu (should open same menu and work in same places as mouse context menu)
    • trees are like desktop: single tab stop, right/left to open/close node, up/down/home/end to move (left moves to parent), enter to activate
    • multi-select tree (like navigator) also has shift+up/down, ctrl+up/down, ctrl+space, shift+space, to manage selection
    • if splitter has focus, up/down or left/right moves splitter, enter to collapse/restore
    • tabFolder with tabItems that are semantic links (i.e. they take you to another page, like in sidebar): use tab key to move through items
    • tabFolder with tabItems that switch to a new tabPanel (like on settings page): arrow keys switch tabPanels; tab key moves focus into tabPanel
    • toolbars are fuzzy:
      • spec says toolbar should be one tab stop and use arrow keys to move through tools (not very discoverable on web)
      • so our toolbars allow tab key to navigate through tools (should consider allowing left/right to move through tools also)
    • tooltips are also fuzzy:
      • spec says tooltips should pop up when trigger element gets focus (doesn't make sense in editor; also would quickly get annoying)
      • so we have decided to open/focus tooltips on F2 and close with esc

Cloud Foundry

This section is all about testing the Cloud Foundry support in Orion. It includes areas around creating launch configurations, setting up cloud preferences, deploying apps, etc.

  • set cloud preferences
  • create an app and launch configuration
  • deploy app

Code Edit Widget

Everything you would ever want to know about the Code edit widget is found in its wiki page.

  • run the widget
  • test the language tools
  • reload / refresh the page
  • edit the eslint / tern-project files

Collaboration

This feature is currently dark launched, and will (obviously) require more than one tester simultaneously.

  • Start collab
  • invite others to your session
  • accept invite to another's session
  • work on same file - editing / location tracked properly
  • add files / folders

Debugging

This feature is currently dark launched.

  • enable the feature
  • create / edit launch configurations
  • show debug views / UI
  • launch simple app
  • attach
  • stepping (over / into / out)
  • stop / disconnect
  • eval hovers

Editor

The editor has many many hidden features. This section is all about testing them (and the obvious ones)

  • open any file
  • js / html / css - validation enabled
  • js / html / css - content assist enabled
  • js / html / css - hovers enabled
  • code preview
  • outliners (view -> side panel)
  • annotation visibility
  • annotation navigation (next / prev annotation)
  • theme
  • auto save
  • tabs
  • cut / copy / paste / line cut
  • format command / auto-format
  • 'typing' prefs - changes behaviour

Git Support

  • navigation to git page from any other page
  • blame in the editor
  • clone repos
  • clone sub-modules
  • pull
  • merge
  • conflicts
  • push
  • unauthenticated handling
  • branching
  • init repo
  • repo list filtering
  • sync command
  • graphical representation
  • view history toggle
  • branch / tag / stash list filtering
  • remotes
  • filter commits
    • various filter options
    • paths
  • working directory
    • amend
    • sign-off
    • credentials
    • expand / collapse
    • diff editor
    • maximize diff editor
    • navigate change in diff (up / down arrows)
    • open diff in editor
    • ignore whitespace
    • open in compare page
  • outgoing changes
    • various push types
    • revert un-pushed commit
    • reset
    • 'more' info
    • create tag
    • show patch
  • incoming changes
    • merge
    • rebase
    • squash
    • various fetch types
    • 'more' information
    • checkout commit
    • reset to commit
    • tag commit
    • cherry-pick commit
    • revert commmit
    • show patch for commit

HTML / CSS Language Tools

  • create new files
  • content assist
  • nodes / classes
  • validation
  • occurrences
  • outlines (side view)
  • refactoring
  • hovers
  • hover navigation

Import / Export

  • import files
  • export files

JS Language Tools

  • create new files
  • content assist
  • validation
  • occurrences
  • all references
  • outlines (side view)
  • hovers
  • hover navigation
  • open decl
  • in-line refactor
  • jsdoc content assist
  • validation prefs
  • .tern-project file support
    • content assist
    • hover
    • validation
    • changes reflected when leaving editor
  • .eslintrc
    • hover
    • validation
    • changes reflected when leaving editor
  • package.json
    • hover
    • validation
    • changes reflected when leaving editor
    • eslintConfig section
  • ECMA 2015
    • "ecmaVersion" must be set to 6 or greater to test these features.
    • classes
    • let, const
    • generators
    • destructuring assignment
    • template literals
    • code templates
    • content assist for ECMA built-ins
    • import / export
      • tooling works across files
      • warns to set project to "module" support vs "script"
  • ECMA 2016
    • "ecmaVersion" must be set to 7 or greater to test these features.
    • array.includes
    • code templates
    • content assist for ECMA built-ins
  • ECMA 2017
    • "ecmaVersion" must be set to 9 or greater to test these features.
    • async / await
    • code templates
    • content assist for ECMA built-ins
  • requirejs
    • tooling works across files
    • detects if require is not running while using
  • node
    • tooling works across files
    • detects if node is not running while using
  • .definitions
    • load custom definition files per-project
    • changes in .definitions folder reflected immediately
  • .ternplugins
    • custom Tern plugins can be loaded per-project
    • changes in .ternplugins reflected immeidately

JSON Language Tools

  • create new files
  • validation
  • hovers
  • hover navigation

Markdown Editor

  • content assist
  • validation
  • preview

Plugins

  • install plugins
  • remove plugins
  • reload plugins
  • plugin details

Preferences

  • navigate to prefs page from anywhere
  • restore defaults immediately (where available)
  • persist across restarts

Search

  • file search
    • all projects
    • project context only
  • in-file search
    • case-sensitive, expression, whole word and replace mode
    • "find in"
    • file name expressions
    • excluded file names
  • in-editor search
    • find / replace
    • options (case sensitive, etc)
    • next / prev match
    • keybinding

Shell

  • open page
  • command completion

Sites

  • navigate to page
  • create new site
  • view existing site
  • make site self-hosting
  • open site
  • delete site
  • rename site
  • start / stop sites

Themes

  • IDE themes
  • editor themes
  • change themes
  • edit / save themes
  • preview themes

Workspace

  • creating folders / files / projects (where supported)
  • resources at workspace root (turn pref on)
  • folder editor pane (details of projects / folders)
  • show problems
  • 'reload with'
  • convert line delimiters
  • keybindings
  • 'open with'
  • 'open related'
  • side panel -> nav or project
  • operations page
    • clean up
    • remove all / selected