Jump to: navigation, search

WTP Performance Tests/2008-09-29

Attendees

Gary Karasiuk Y
Angel Vera
Mark Hutchinson Y
Nick Sandonato
Carl Anderson Y
Kaloyan Raev Y
Raghu Srinivasan
Neil Hauge Y
David Williams Y

Agenda

Has every team identified a single test case?
Do we have success on running the performance test procedure on the a single test?
Best practices for writing Eclipse performance tests

Minutes

Has every team identified a single test case?
Gary promised to make it for the Common Tools team by next week. The other team with pending test case is JSF Tools.
Do we have success on running the performance test procedure on the a single test?
Nick sent a very useful information to Kaloyan. Now he is able to do a successful manual run of the graph generation tool.
Still have problems with the automatic procedure - with the CVS download of the graph generation tool.
The Ant scripts need refactoring to make it easy for automatically execute some main tasks:
  • set a baseline - not possible at the moment without manually manipulating the DB
  • run a test and compare against a baseline - almost possible to do this now, but still needs some optimization in terms of where the scripts downloads tools and plugins. For example, the org.eclipse.test.performance plugin is downloaded from the already released build and have to be patched manually.
Best practices for writing Eclipse performance tests
We need to discuss what exactly do we want to measure.
Kaloyan mentioned that we should respect the JIT compiler optimizations
Gary expressed an opinion that we should measure big things - a complete user tasks, like executing a build, finishing a wizard, opening big files in an editor, etc.
David said that we should also take into account some small operations (less than a second) that are executed repeatedly in bigger operations or a critical to execute quickly. We should have to types of performance tests like:
  • assurance tests - for major user tasks that take significant time
  • diagnostic tests - for tiny operation that are critical to execute quickly
All thoughts should be captured in the Best practices section of the main performance tests wiki page.
Other thoughts
We should summarize all our open tasks and see if we are able to handle them in the short term.
In any case we must document all our findings.

Action items

Refactor Ant scripts (open)
Summarize latest findings (Kaloyan)
Write thoughts about best practices and priorities for measurement (Gary and others)
Summarize the list of open action items (Kaloyan)

References