Difference between revisions of "Talk:Aperi/Test/Test Plans/IDVT"

Revision as of 22:54, 25 October 2006

Aperi IDVT Test Plan - Phase 2

Document Approvers (Please add "I Approve" followed by your signature below your name to indicate your approval of the test plan)

Ted Slupesky

Chris Rich

Mandatory Document Reviewers

Dave Wolfe

Tom Guinane

Hans Lin

Comments

Ted Slupesky:

This looks good, although I have some comments that you should respond to before I can approve it.

- You should assume that the 'new' GUI will be built as part of the nightly build, and installable in the usual way, in time for your testing to begin.

- I think section 1.7 should be deleted -- or I don't understand why it's there and empty. The information I thought would be there is in section 3.1.1.

- Specific references to IBM and TPC should be removed (e.g. "Source: TPC 3.1 IDVT Plan")

- We can go straight to the external wiki for this, no reason to use the internal one (once the IBM/TPC references are removed).

- On that note, you should remove the existing title page and document control information (everything up to the table of contents) and replace it with a new title page -- no references to Tivoli, IBM Confidential, or anything like that. For an example see http://www.eclipse.org/aperi/tech/Aperi_System_Design.doc

- It would be swell if this wasn't MS Word. If it stays MS Word, then I don't know how to put it on the wiki portion of our eclipse site -- but I can show you a directory I created elsewhere (not in the wiki) you can use to check it in. You are a committer, right?

Brian's response to Ted's comments:

"You should assume that the 'new' GUI will be built as part of the nightly build, and installable in the usual way, in time for your testing to begin." - OK, included this as an entry criterion

"I think section 1.7 should be deleted -- or I don't understand why it's there and empty. The information I thought would be there is in section 3.1.1." - Agreed, deleted section 1.7

"Specific references to IBM and TPC should be removed" - Removed all references to IBM and TPC

"We can go straight to the external wiki for this, no reason to use the internal one (once the IBM/TPC references are removed)." - Will use the external wiki to post status reports

"On that note, you should remove the existing title page and document control information (everything up to the table of contents) and replace it with a new title page -- no references to Tivoli, IBM Confidential, or anything like that." - Changed the title page, removed the document control information, and removed all references to Tivoli, IBM Confidential

"It would be swell if this wasn't MS Word." - Changed the file format to pdf.

Hans Lin:

Environmental Requirements This section lists both the necessary and desired properties required for the test environment. Hardware Two Windows servers to host APERI server components -Why do we need two Windows machines to host APERI server components? ( One for Data and One for Device ) ?

One Windows machine running the Data agent (and not running the server) -I think it will make more sense to have two windows machines for agents ( one with HBA and one without HBA to host Fabric agent and Data agent). At least, we need a windows machine ( with HBA ) to have both data and fabric agents installed

One Linux machine running APERI server and agent components One NAS server One tape device At least one subsystem visible to test environment Brocade switch McData switch Cisco switch A large number of storage subsystems (desired for performance testing) Storage subsystems with a large number of volumes (desired for performance testing) A large number of hosts (desired for performance testing)

Install and Configuration Tests Source: Initial Contribution IDVT installation and configuration for the Initial Donation will be done using the Aperi build procedure and will be used to verify that the Data Server, Device Server, Data Agent, database, and GUI are all configured and started without errors.

-We need to include Fabric agent

Fabric Alerts and Alert Logs

-We didn't have Disk Alerts testcases for our Phase I IDVT, I think we should do it for our phase II IDVT Specific Views Include testing of one example of each kind of view: SAN L0 SAN L1 SAN L2 Computer L0 Computer L1 Computer L2

-We should include the entire Topology views as follows L0:Computers L1:Computers-status L2:Computer-computer L0:Fabrics L1:Fabric-fabric L2:Switch-switch L0:Storage L1:Subsystems-status L2:Subsystem-subsystem L0:Other L1:Others-status L2:Other-other

In our Phase I donation, we didn't include Fabric agent support so there is no zoning support for Cisco and McData switches. We should have some zone test cases for out Phase II IDVT

Brian's response to Han's comments:

"Why do we need two Windows machines to host APERI server components?" - The use of multiple servers is only to use testers more effciently

"I think it will make more sense to have two windows machines for agents" - Agreed, changed to two windows machines, one with an hba and one without

"We need to include Fabric agent" - Agreed, changed wording to include the Fabric agent

"We didn't have Disk Alerts testcases for our Phase I IDVT, I think we should do it for our phase II IDVT" - Agreed, added test cases for disk alerts

"We should include the entire Topology views" - Added suggested topology view test cases

"In our Phase I donation, we didn't include Fabric agent support so there is no zoning support for Cisco and McData switches. We should have some zone test cases for out Phase II IDVT" - Added scenarios to test support for zoning on Cisco and McData switches

Tom Guinane:

Front page - need to remove 'Tivoli', 'IBM', 'Confidential', etc. In fact, do a scan and get rid of anything like that in the entire document. For this document, we are Eclipse and specifically, the Aperi Storage Management Project.

page 2 - If you did this as a wiki page, you could remove the document control information from here and only list in the 'discussion' section. You could also reference the document source directly (the URL for the page)

page 4, Introduction - Delete the words "and before coding begins for Phase 3" out of the second sentence. It is unnecessary and is not what will really happen.

page 5, Section 1.3, last sentence - we will be using Bugzilla at Eclipse, not CMVC to track defects. I'm not sure how we tell which defects are ours in Bugzilla. There must be some code similar to CMVC. We need to check that out.

page 5, Section 1.4 - need to add "The IDVT Test Plan has been approved" as the first entry criteria for the first part of IDVT.

page 5, Section 1.4, part two - need to fix the wording of the second bullet.

page 5, Section 1.5, bullet two - Any particular reason we are deviating from our normal 95% success rate with no sev 1's or 2's? This leaves us wide open for a success rate.

page 5, Section 1.4, bullet three - To exit test, we need to have objective measurement evidence. How do you plan to measure that for this item? Just the existence of it? What about the quality of this item? This might be a difficult to validate.

page 7, 2.1.1 - I like this. Just remember, as it is a test entry criteria, you will need to provide evidence for each of these.

page 8, 2.1.2 - performance numbers at the top - should these be listed in this external document?

page 9, 3.1.4, GUI - this is good. We will need to have the plan amended for the GUI tests before that testing begins.

page 11, 4.1 - I think the external wiki is fine for this. Will we have S-curves (like we did in Phase 1)?

page 12, 5.1.3 - GUI date will probably move out to 3/16/07 with the GUI changes.

Section 5, most pages - remove TPC references from Source areas

Brian's response to Tom's comments:

"Front page - need to remove 'Tivoli', 'IBM', 'Confidential', etc...." - Ok, removed all references to 'Tivoli', 'IBM', 'Confidential', etc.

"page 2 - If you did this as a wiki page, you could remove the document control information..." - OK, removed document control page

"page 4, Introduction - Delete the words "and before coding begins for Phase 3" out of the second sentence. It is unnecessary and is not what will really happen." - Done

"page 5, Section 1.3, last sentence - we will be using Bugzilla at Eclipse, not CMVC to track defects..." - Changed wording to Bugzilla instead of CMVC

"page 5, Section 1.4 - need to add "The IDVT Test Plan has been approved" as the first entry criteria for the first part of IDVT." - Done

"page 5, Section 1.4, part two - need to fix the wording of the second bullet." - Done

"page 5, Section 1.5, bullet two - Any particular reason we are deviating from our normal 95% success rate..." - Actually, what I meant was that we will not exit if we have sev 1's or sev 2 defects. Changed the wording to reflect this and added another criterion that reflects a 95% success rate in order to exit.

"page 5, Section 1.4, bullet three - To exit test, we need to have objective measurement evidence..." - The acceptance test is an entry criterion. Its purpose is to ensure that basic functions of the Aperi Storage Manager is working and that the software is in a stable enough condition to enter IDVT. Details of the acceptance test are listed in the test plan.

"page 7, 2.1.1 - I like this. Just remember, as it is a test entry criteria, you will need to provide evidence for each of these." - Yes, I intend to provide evidence that validates the tests detailed in section 2.1.1.

"page 8, 2.1.2 - performance numbers at the top - should these be listed in this external document?" - I removed the numbers and reworded the performance issues

"page 9, 3.1.4, GUI - this is good. We will need to have the plan amended for the GUI tests before that testing begins." - Will do

"page 11, 4.1 - I think the external wiki is fine for this. Will we have S-curves (like we did in Phase 1)?" - Removed wording that indicated the use of the internal wiki. All status reports will be placed on the external wiki, including S-curve charts detailing the progression of test.

"page 12, 5.1.3 - GUI date will probably move out to 3/16/07 with the GUI changes." - Modified the completion date for testing the 'new' GUI to 3/16/07

"Section 5, most pages - remove TPC references from Source areas" - Done

Chris King:

I looked over the plan and have some minor suggestions.

You might want to add test cases for the online help in the new Eclipse RCP-based user interface. This interface will have an accompanying online help system using the new Eclipse framework and should work like the existing help system in legacy user interface (i.e., press F1 to access the help).

For example:

- Ensure that the correct help topic is appearing for the product window that invoked the help. For example, press F1 from one of the generated BIRT reports (Disk Capacity > By Disk) included in the new interface and press F1 to confirm the correct help topic appears.

- Check the links between the entries in the help's table of contents and the actual topics in the help. For example, click on an entry in the help's table of contents (left pane) and confirm that the corresponding help topic appears (in the right pane).

- Verify links from one help topic to another. For example, confirm that the link in the Disk Capacity > By Disk help window correctly links to the help topic that contains overview help for all reports.

- Test the Back, Forward, and print buttons in the tool/menu bar.

- Test the cheat sheets (if any): Ensure that the correct cheat sheet appears for the product window, check any links within the cheat sheet to confirm that they link correctly, etc.

Brian's response to Chris's comments:

Added test cases to cover each of Chris's suggestions, with the exception of one - "Test the Back, Forward, and print buttons in the tool/menu bar." - According to developer Bill Warren printing to printers, pdf's, etc. was pulled out of Aperi. Aperi will only be able to export to html and print charts to html. That's because the old printing support depended on stuff in JChart, and we didnt have time to replace that right away.

Dave Wolfe:

The scope with respect to the RCP GUI needs some update as the plans are changing.

The R2 and R3 topology viewer is a new code base merged from work at IBM research and will incorporate a lot of schema changes to improve scaling and performace of the troublesome queries. This also includes some new functionality with respect to SVC.

We are planning on fully encapsulating the Legacy GUI into the RCP GUI as new perspective/view. This encapsulated Classic View will replace the Legacy GUI in the final product (at R3). 99.9% of the code will be unchanged, so testing completed on the Legacy GUI will not be invalidated, but some level of regression testing will be required. Along with this change, the RCP based topology viewer moves out of R3 as a formal deliverable (since there will be a toplogy viewer in the classic view). So, the key new functionality to focus on for the R3 release will be the BIRT based report viewer and the handful of reports that will be implemented at that time.

In section 1.6 you mention (externally visible) CLI. I don't think we have any of those in Aperi.

In section 5.1.4.7, the bullet Printing and Print Review - as I recall we pulled the printing subsystem out of Aperi sicne it was absed on proprietary technology. You can still 'print' to HTML but not to a printer (unless I am remembering wrong). You should double check with Bill Warren on that.

Brian's response to Dave's comments:

"The scope with respect to the RCP GUI needs some update as the plans are changing." - We realize that the design of the GUI is an ongoing process. We intend to continually update the test report and test cases so that they are aligned with changes that occur to the GUI

"We are planning on fully encapsulating the Legacy GUI into the RCP GUI as new perspective/view..." - That is good news. This means that we can probably base our automated tests on the legacy GUI, since it will remain relatively constant. I will update the test report to reflect our intention.

"In section 1.6 you mention (externally visible) CLI. I don't think we have any of those in Aperi." - We do use the cli to install and configure Aperi, therefore, I will leave this statement in the report

"In section 5.1.4.7, the bullet Printing and Print Review - as I recall..." - You are right. I have removed the print test cases from the test plan