Use the Parameter Editor to create and edit configuration parameters. These parameters can also be created and changed using the Configuration Editor. The JavaTest harness grays out any parameters that are not used by your test suite.
The content pane contains two areas where you can specify the tests that are run:
Restrictions are applied cumulatively. For example, you can restrict the set of tests using an exclude list and then further restrict the set to only those tests that passed on a prior run.
Create a list of test directories and/or tests that you wish to run. The JavaTest harness walks the test tree starting with the sub-branches and/or tests you specify and executes all tests that it finds (unless they are filtered out).
Add | To create the list of directories and tests in your test suite, click Add. As you make selections with the file chooser dialog box, the JavaTest harness adds them to the list. After you have added selections, you can modify the list. |
Remove | To clear an item from the list, select it in the list and click Remove. |
Move Up | To move an item one position higher in the list, select it in the list and click Move Up. |
Move Down | To move an item one position lower in the list, select it in the list and click Move Down. |
The JavaTest harness uses four types of filters to include or exclude tests in a test run. The first filter is a hidden filter optionally provided by the test suite. The JavaTest harness automatically uses it when provided by the test suite. The other three filters are:
Exclude tests from a test run by specifying one or more exclude list files. An exclude list file contains a list of the conformance tests in a test suite that are not required to be passed to certify conformance.
Add | To select an exclude list file for your test suite, click Add. As you make selections with the file chooser dialog box, they are added to the list. After you have added an exclude list, you can modify the list. |
Remove | To clear an item from the list, select it in the list and click Remove. |
Move Up | To move an item one position higher in the list, select it in the list and click Move Up. |
Move Down | To move an item one position lower in the list, select it in the list and click Move Down. |
You can use the combo box and text field to restrict the set of tests to be run based on keywords associated with the tests in the test suite:
ignore | The default selection. Keyword filtering is not used. | |
any of |
Runs all tests in the test suite having any of the keywords entered in the text field.
Example: A test suite uses the keyword "interactive" to identify tests that require human interaction, and "color" to identify tests that require a color display. Choose any of from the combo box and then enter the following in the text field to execute only those tests containing the "interactive" keyword.
|
|
all of |
Runs all tests in the test suite having all of the keywords entered in the text field.
Choose all of from the combo box and then enter the following in the text field to execute only those tests containing both the "interactive" and "color" keywords.
|
|
expr |
Runs all tests in the test suite having the expression entered in the text field
In the text field, construct a boolean expression, built up from keywords and the operators "&" (and), "!" (not), "|" (or), and "( )" (parentheses for grouping). Keywords stand as boolean predicates that are true if, and only if, the keyword is present in the test being considered. A test is accepted if the overall value of the expression is true; all other tests are rejected by the restriction. Example: A test suite uses the keyword "interactive" to identify tests that require human interaction, and "color" to identify tests that require a color display. Choose expr from the combo box and then enter the following expression to execute only those tests with the "color" keyword that do not also contain the "interactive" keyword.
|
You can use the combo box and check boxes to select tests in a test run based on their outcome on a prior test run:
ignore | The default selection. Prior status filtering is not used. | |||||||||
any of | Runs all tests in the test suite having any of the following status values you choose to use as filters during test runs: | |||||||||
|
It is often useful to choose all of the status values except "passed" for the first few test runs, then refine the filtering to reduce the number of tests in subsequent runs.
Prior status is evaluated on a test-by-test basis using information stored in result files (.jtr
) written in the work directory. Unless overridden by a test suite, a result file is written in the work directory for every test that is executed. If you change the work directory between test runs, the result files will not be found. If the new work directory is empty, the JavaTest harness behaves as if the test suite was never run.
Contains four fields where you can specify how the tests are run:
Environment files contain test environments used to run tests in your computing environment. You can create or modify the list of environment files.
The JavaTest harness may disable this field if it is not supported by your test suite.
Add |
To select an environment file (.jte ) for your test suite, click Add. As you make selections with the file chooser dialog box, they are added to the list. After you have added an environment file, you can modify the list.
|
Remove | To clear an item from the list, select it in the list and click Remove. |
Move Up | To move an environment file one position higher in the list, select it in the list and click Move Up. |
Move Down | To move an environment file one position lower in the list, select it in the list and click Move Down. |
Click the drop-down arrow on this field to see the list of test environments in the environment file(s) listed in the Env Files list. Click the Show button to view the contents of the selected test environment.
The JavaTest harness may disable this field if it is not supported by your test suite.
The JavaTest harness can run tests concurrently. If you are running the tests on a multi-processor computer, concurrency can speed up your test runs.
However, some test suites may not work correctly if you run tests concurrently. If you have unexpected test failures, run the tests again one at a time.
For your first test run you should leave this field blank or set to "1" (which is equivalent to blank). After you have the tests running properly you can experiment with this value.
To prevent a stalled test from stopping a test run, most test suites use a timeout limit for each test. The timeout limit is the amount of time that the JavaTest harness waits for a test to complete before moving on to the next test.
If you are running the tests on a particularly slow CPU or slow network, you can increase the time limit by specifying a value in the time factor field. Each test's timeout limit is multiplied by the time factor value.
Example:
If you specify a value of "2", the timeout limit for tests with a 10 minute basic time limit becomes 20 minutes.
Try running without a time factor first and increase the value as necessary.
The content pane contains a Report Directory field you can use to specify the location where the JavaTest harness writes the test reports.
Specify the directory where the test report files are written. If this path is not specified, the reports are written to a directory named report
in the work directory used to run the tests.
After the test run is completed, the JavaTest harness writes HTML reports for the test run. You can view the reports in a web browser:
The report.html file is the root file that links to all of the other HTML reports. The JavaTest harness also creates a summary.txt file containing a list of all tests that were run, their test results, and their status messages.
When the JavaTest harness writes new test result files, it backs up the previous test report file names with the test run number so that the results of the previous runs remain available for review. Currently, the JavaTest harness maintains a maximum of five backups in the report directory.