Jump to: navigation, search

Difference between revisions of "PDE/API Tools/Testing"

< PDE‎ | API Tools
(Quick Fixes)
(Test Script)
 
(15 intermediate revisions by 3 users not shown)
Line 3: Line 3:
 
== Test Page Template  ==
 
== Test Page Template  ==
  
Test Page Template: [[PDE/API Tools/Test Plan Template]].  
+
[[PDE/API Tools/Test Plan Template|Test Page Template]].  
  
 
This template must be used to create a new test page for each release that requires testing.  
 
This template must be used to create a new test page for each release that requires testing.  
  
Any new features that are added to the [[PDE/API Tools/Test Script|test script]] must also be added to the [[PDE/API Tools/Test Plan Template#Feature_List|feature list]] section of the template.  
+
Any new features that are added to the [[PDE/API Tools/Testing#Test_Script|test script]] must also be added to the [[PDE/API Tools/Test Plan Template#Feature_List|feature list]] section of the template.
  
 
== Test Passes  ==
 
== Test Passes  ==
  
 +
*[[PDE/API Tools/Test Plan ASM5|ASM 5.0 Upgrade]] (February 25, 2014)
 
*[[PDE/API Tools/Test Plan 3.7M7|3.7 M7]] (April 29, 2011)
 
*[[PDE/API Tools/Test Plan 3.7M7|3.7 M7]] (April 29, 2011)
  
== API Setup  ==
+
== Test Script ==
 +
=== API Setup  ===
  
 
'''General'''  
 
'''General'''  
Line 28: Line 30:
 
*ensure tags are added correctly to files (from the component.xml)
 
*ensure tags are added correctly to files (from the component.xml)
  
== API Baselines  ==
+
=== Annotations ===
 +
'''General'''
 +
*Confirm the API description file is the same as with Javadoc tags
 +
*Confirm the API file generation task outputs the same description as it did with the Javadoc tags
 +
 
 +
'''Content Assist'''
 +
*Use the API annotations in source
 +
*Confirm content assist offers them
 +
*Confirm the quick-fix to set up you bundle for them
 +
 
 +
'''Conversion'''
 +
*Use the conversion wizard
 +
*Confirm the wizard adds them in source
 +
*Confirm the wizard optionally removes old Javadoc tags
 +
 
 +
'''Validation'''
 +
*Confirm the validation preferences work
 +
*Confirm the annotations are validated: no dupes, location, usage
 +
 
 +
=== API Baselines  ===
 
'''General'''
 
'''General'''
 
*changing / editing / removing of the default baseline should ask you to build when the Ok button is pressed on the preference page the page and wizard should have help topics
 
*changing / editing / removing of the default baseline should ask you to build when the Ok button is pressed on the preference page the page and wizard should have help topics
Line 48: Line 69:
 
'''Change the warning level for missing default baselines'''
 
'''Change the warning level for missing default baselines'''
  
== Building  ==
+
=== Building  ===
 
'''Full Build'''
 
'''Full Build'''
*Run a full build on one project
+
*run a full build on one project
 
*markers should be updated
 
*markers should be updated
 
*filters should be respected
 
*filters should be respected
Line 62: Line 83:
 
*time to build should be reasonable
 
*time to build should be reasonable
 
'''Incremental Building'''
 
'''Incremental Building'''
*Change a dependent type
+
*change a dependent type
 
*builder should run for dependent's
 
*builder should run for dependent's
 
*markers should be updated for dependent's
 
*markers should be updated for dependent's
Line 68: Line 89:
 
*time to build should be reasonable
 
*time to build should be reasonable
  
== Unsupported Javadoc Tags  ==
+
=== Unsupported Javadoc Tags  ===
 
Enable the check for unsupported Javadoc tag checking. All of the given tests should produce API problems, and all content assist-available tags should not
 
Enable the check for unsupported Javadoc tag checking. All of the given tests should produce API problems, and all content assist-available tags should not
  
Line 78: Line 99:
 
*add noimplement, noextend, noreference, nooverride and noinstantiate tags to a final field
 
*add noimplement, noextend, noreference, nooverride and noinstantiate tags to a final field
  
== Quick Fixes  ==
+
=== Quick Fixes  ===
 
All quick fixes should not put a project in a state of error, and should work with an existing modified editor (i.e. even if there are unsaved edits in the editor the quick-fix is being activated on)
 
All quick fixes should not put a project in a state of error, and should work with an existing modified editor (i.e. even if there are unsaved edits in the editor the quick-fix is being activated on)
  
*Usage problems
+
*usage problems
*Compatibility problems
+
*compatibility problems
*Version problems
+
*version problems
*Since Tag problems
+
*since tag problems
*API Leak problems
+
*API leak problems
 +
 
 +
=== Filters  ===
 +
'''General'''
 +
*property page should have a help topic
 +
*remove button should enable / disable properly
 +
'''Add Filters'''
 +
*use the quick fix (and the commenting quick-fix)
 +
*drag & Drop in an existing filter file
 +
*check out a filter file from CVS
 +
*first added filter should cause filter file to be created immediately
 +
*other added filters should be written to filter file immediately
 +
*add / remove comments from a filter
 +
'''Remove Filters'''
 +
*use the property page
 +
**try removing one or more filters by individually selecting filters or their parent file.
 +
**ensure removed filters are removed (apply change or click Ok to close the property page, and reopen)
 +
**canceling deleting filters should make no change
 +
**removing filters should rebuild the project (incremental)
 +
**removing last filter should remove the filter file immediately
 +
*drag & Drop a filter file out of the project
 +
*over-ride and update from CVS
 +
*auto-remove using the builder preference
 +
 
 +
A good test case: (1) start new workspace, (2) bring org.eclipse.ui.navigator from R3_5 branch (3) set baseline to an Eclipse 3.5.0 install.
 +
 
 +
With the above set-up now change the setting in the workspace and the project and ensure filters are automatically cleaned up or reported depending on the setting.
 +
 
 +
'''Share Filters'''
 +
*commit filter to CVS and check it out in another workspace
 +
*should update filtered problems on next rebuild (unless the filter store for the project in question is already loaded)
 +
'''Unused Filters'''
 +
*create a filter for a Java element and them remove the element - problem should be reported
 +
*create a filter for a Java element and delete the containing compilation unit - no problem (filters are cleaned up when the filter store is initialized / workspace restarted)
 +
*quick-fix to remove the filter
 +
*quick-fix to open the property page
 +
 
 +
=== Preferences  ===
 +
'''General'''
 +
*the API Errors/Warnings page should have a help topic
 +
'''Workspace'''
 +
*change any number of preferences and apply the page (via the Apply button or the Ok button)
 +
*change any number of preferences and hit 'Restore Default'
 +
*change any number of preferences and cancel the page
 +
*change all preferences using the 'Set all to' buttons (and apply / cancel / restore defaults)
 +
*use the link to configure project specific settings
 +
'''Per-Project'''
 +
*try turning project specific settings on / off
 +
*change any number of preferences and apply the page (via the Apply button or the Ok button)
 +
*change any number of preferences and hit 'Restore Default'
 +
*change any number of preferences and cancel the page
 +
*change all preferences using the 'Set all to' buttons (and apply / cancel / restore defaults)
 +
*use the link to configure workspace settings
  
== Filters ==
+
=== Invalid System Library References ===
 +
*change EE to 1.4 (or prior), set build-path JRE to be 1.5 or greater, and access types / methods / fields from the build path library
 +
* add / remove EE descriptions from the preference page
  
== Preferences ==
+
=== Target Platform Stability Workspace Baseline ===
 +
These tests are to ensure there is no corruption to the target platform when performing workspace baseline changes during normal development. All of the following tests require that the Target Platform State view be open and that it be checked after each of the following tests to ensure no bundle problems arise.
  
== Invalid System Library References  ==
+
*create / delete projects in your workspace
 +
*bring in projects from CVS
 +
*open / close projects
 +
*import projects (binary and normal projects)
  
== Target Platform Stability Workspace Baseline ==
+
=== Use Scan Reference ===
 +
'''API Use Scans'''
 +
*Test adding directory and archived API Use Scans
 +
*Test with adding invalid use scans (directory and archives)
 +
*Test Edit/Remove
 +
*Test that build is triggered after changes
 +
'''Use Scan Preferences'''
 +
*Test setting preferences on API Use Scans tab of API Error/Warnings
 +
*Test with different severities for Type, Method and Field problems
 +
*Test that severity changes are reflected in markers in Problems view
 +
'''Ant Task'''
 +
*Test running the apitooling.apiusescanproblems ant task
 +
*Test that it generates same problems as in the problems view
 +
'''References'''
 +
*Test for missing Type, Method and Field
 +
*Test that missing method marker is placed at correct line numbers and are made on proper java file
 +
*Test that missing field marker is placed at correct line number and are made on proper java file
 +
*Test that missing type markers has line number as -1 and are made on Project
  
== Use Scan Reference  ==
+
=== Run The Test Suite ===
 +
The shared launch configurations in the <code>org.eclipse.pde.api.tools.tests</code> bundle can be used to run these
  
== Run The Test Suite ==
+
*run ApiToolsTestSuite
 +
*run ApiToolsPluginTestSuite

Latest revision as of 10:49, 25 February 2014

API Tools
Website
Download
Community
Mailing ListForumsIRC
Bugzilla
Open
Help Wanted
Bug Day
Contribute
Browse Source

Test Page Template

Test Page Template.

This template must be used to create a new test page for each release that requires testing.

Any new features that are added to the test script must also be added to the feature list section of the template.

Test Passes

Test Script

API Setup

General

  • setup wizard should have a help topic

Setup API Tooling

  • wizard with no selections in workspace and with selections; if the there are selected projects in the workspace that can have API tooling setup on them, they should be preselected in the wizard
  • try canceling the wizard
  • try selecting / unselecting all to make sure the finish and preview button work as expected</li>
  • try selecting a project and viewing a preview
  • try selecting projects that do and do not have a component.xml file
  • ensure tags are added correctly to files (from the component.xml)

Annotations

General

  • Confirm the API description file is the same as with Javadoc tags
  • Confirm the API file generation task outputs the same description as it did with the Javadoc tags

Content Assist

  • Use the API annotations in source
  • Confirm content assist offers them
  • Confirm the quick-fix to set up you bundle for them

Conversion

  • Use the conversion wizard
  • Confirm the wizard adds them in source
  • Confirm the wizard optionally removes old Javadoc tags

Validation

  • Confirm the validation preferences work
  • Confirm the annotations are validated: no dupes, location, usage

API Baselines

General

  • changing / editing / removing of the default baseline should ask you to build when the Ok button is pressed on the preference page the page and wizard should have help topics

Create a new API baseline

  • try completing the wizard and canceling it
  • try renaming a baseline (accepting and canceling the process)
  • try changing the location of the new baseline and resetting the profile

Edit an API baseline

  • try completing the wizard and canceling it
  • try renaming a baseline (accepting and canceling the process)

Deleting an API baseline

  • try deleting a baseline
  • try deleting select baselines
  • try deleting all baselines

Default Baseline

  • try selecting a new default baseline
  • try editing the default baseline (rename, should still be the default after rename)
  • try deleting the default baseline

Change the warning level for missing default baselines

Building

Full Build

  • run a full build on one project
  • markers should be updated
  • filters should be respected
  • progress messages should be correct
  • time to build should be reasonable

Run a full build on the workspace

  • builder should only run on 'API aware' projects
  • markers should be updated
  • filters should be respected
  • progress messages should be correct
  • time to build should be reasonable

Incremental Building

  • change a dependent type
  • builder should run for dependent's
  • markers should be updated for dependent's
  • filters should be respected on type and dependent's
  • time to build should be reasonable

Unsupported Javadoc Tags

Enable the check for unsupported Javadoc tag checking. All of the given tests should produce API problems, and all content assist-available tags should not

  • add noimplement, no reference and nooverride tags to class (inner, outer, static, static inner, etc)
  • add noinstantiate, noextend, noreference and nooverride tags to an interface (annotation)
  • add noimplement, noextend and noinstantiate tags to a method
  • add noimplement, noextend, noreference and noinstantiate tags to a constructor
  • add noimplement, noextend, nooverride and noinstantiate tags to a field
  • add noimplement, noextend, noreference, nooverride and noinstantiate tags to a final field

Quick Fixes

All quick fixes should not put a project in a state of error, and should work with an existing modified editor (i.e. even if there are unsaved edits in the editor the quick-fix is being activated on)

  • usage problems
  • compatibility problems
  • version problems
  • since tag problems
  • API leak problems

Filters

General

  • property page should have a help topic
  • remove button should enable / disable properly

Add Filters

  • use the quick fix (and the commenting quick-fix)
  • drag & Drop in an existing filter file
  • check out a filter file from CVS
  • first added filter should cause filter file to be created immediately
  • other added filters should be written to filter file immediately
  • add / remove comments from a filter

Remove Filters

  • use the property page
    • try removing one or more filters by individually selecting filters or their parent file.
    • ensure removed filters are removed (apply change or click Ok to close the property page, and reopen)
    • canceling deleting filters should make no change
    • removing filters should rebuild the project (incremental)
    • removing last filter should remove the filter file immediately
  • drag & Drop a filter file out of the project
  • over-ride and update from CVS
  • auto-remove using the builder preference

A good test case: (1) start new workspace, (2) bring org.eclipse.ui.navigator from R3_5 branch (3) set baseline to an Eclipse 3.5.0 install.

With the above set-up now change the setting in the workspace and the project and ensure filters are automatically cleaned up or reported depending on the setting.

Share Filters

  • commit filter to CVS and check it out in another workspace
  • should update filtered problems on next rebuild (unless the filter store for the project in question is already loaded)

Unused Filters

  • create a filter for a Java element and them remove the element - problem should be reported
  • create a filter for a Java element and delete the containing compilation unit - no problem (filters are cleaned up when the filter store is initialized / workspace restarted)
  • quick-fix to remove the filter
  • quick-fix to open the property page

Preferences

General

  • the API Errors/Warnings page should have a help topic

Workspace

  • change any number of preferences and apply the page (via the Apply button or the Ok button)
  • change any number of preferences and hit 'Restore Default'
  • change any number of preferences and cancel the page
  • change all preferences using the 'Set all to' buttons (and apply / cancel / restore defaults)
  • use the link to configure project specific settings

Per-Project

  • try turning project specific settings on / off
  • change any number of preferences and apply the page (via the Apply button or the Ok button)
  • change any number of preferences and hit 'Restore Default'
  • change any number of preferences and cancel the page
  • change all preferences using the 'Set all to' buttons (and apply / cancel / restore defaults)
  • use the link to configure workspace settings

Invalid System Library References

  • change EE to 1.4 (or prior), set build-path JRE to be 1.5 or greater, and access types / methods / fields from the build path library
  • add / remove EE descriptions from the preference page

Target Platform Stability Workspace Baseline

These tests are to ensure there is no corruption to the target platform when performing workspace baseline changes during normal development. All of the following tests require that the Target Platform State view be open and that it be checked after each of the following tests to ensure no bundle problems arise.

  • create / delete projects in your workspace
  • bring in projects from CVS
  • open / close projects
  • import projects (binary and normal projects)

Use Scan Reference

API Use Scans

  • Test adding directory and archived API Use Scans
  • Test with adding invalid use scans (directory and archives)
  • Test Edit/Remove
  • Test that build is triggered after changes

Use Scan Preferences

  • Test setting preferences on API Use Scans tab of API Error/Warnings
  • Test with different severities for Type, Method and Field problems
  • Test that severity changes are reflected in markers in Problems view

Ant Task

  • Test running the apitooling.apiusescanproblems ant task
  • Test that it generates same problems as in the problems view

References

  • Test for missing Type, Method and Field
  • Test that missing method marker is placed at correct line numbers and are made on proper java file
  • Test that missing field marker is placed at correct line number and are made on proper java file
  • Test that missing type markers has line number as -1 and are made on Project

Run The Test Suite

The shared launch configurations in the org.eclipse.pde.api.tools.tests bundle can be used to run these

  • run ApiToolsTestSuite
  • run ApiToolsPluginTestSuite