TestRunner

The execution of a test run is the job of the TestRunner. The TestRunner is started from the command prompt and configured via a configuration file in xml format. The call from the command prompt is as follows:

 java -jar [instdir]/ivalidator.jar [config]

[instdir] is the installation directory of the iValidator release.

[config] is the configuration file. This parameter is optional. If it is omitted the TestRunner expects a file named configuration.xml in the current directory. If this file is not available the following default settings apply:

<?xml version="1.0" encoding="UTF-8"?>

    <!-- default xml based configuration -->	

<config xmlns="http://www.ivalidator.org/schemas/ivalidator" 
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<descriptor repository="xml">
		<property name="input" value="test.xml"/>
		<property name="classpath" value="./;./bin/;./classes/"/>
	</descriptor>

	<report repository="xml">
		<property name="outputdir" value="./"/>
	</report>

</config>

The configuration can be specified in relativ or absolute form. Relativ declarations refer to the current directory. If only a directory is given the file name is expected to be configuration.xml. The declaration can also be an URL.

Configuration

Configuration File

The test configuration contains all necessary information concerning the repositories for the test run.The structure of the configuration file must satisfy the w3c-schema configuration.xsd.

The default name is configuration.xml The configuration file may contain the following properties for the iValidator.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://www.ivalidator.org/schemas/ivalidator" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<!-- Framework Configuration-->	
	<property name="ivalidator.junitinits" value="false"/>
	<property name="ivalidator.parentparamsfirst" value="true"/>
	
	<!-- Test Description -->
	<descriptor repository="xml">
		<property name="inputpath" value="input"/>
		<property name="input" value="testsuite_simple.xml"/>
		<property name="classpath" value="classes"/>
	</descriptor>

	<descriptor repository="xml">
		<property name="inputpath" value="input"/>
		<property name="input" value="testsuite_more.xml"/>
		<property name="classpath" value="classes"/>
	</descriptor>

	<!-- Test Results -->
	<report repository="xml">
		<property name="outputdir" value="result"/>
	</report>
	
</config> 

The element <descriptor> is used to specify which test descriptor will be executed. The element <report> is used to specify where the test results are stored. Both elements have the same structure. Basically a repository type has to be provided. The repository can then be configured via propertiery. Which properties a repository expects depends on the repository and must be documented there.

Descriptors as well as reports can be specified repeatedly. In this case all the descriptor-repositories will be executed and all results are written to all specified report-repositories.

System Properties

The iValidator properties of the configuration files can alternatively also be specified as vm-arguments when starting the iValidator. Additionally the following system properties can be set exclusivly as vm-arguments.

Test and Validation Run

Validation Run

Before the actual test run a so-called validation run takes place. The validation run usally only takes a few seconds and is used to find errors in the test description that might lead to an abortion of a long running test run.

The validation run checks two things. The first is, to see whether the specified test classes are available and can be instantiated.

Furthermore the parameters from the test descriptors are checked. For that check it is necessary that a unit defines which parameters it requires. For example:

     public ParameterDescription[] paramTestLoginSuccessful()
    {
        return new ParameterDescription[] {
                new ParameterDescription("username", String.class, true),
                new ParameterDescription("password", String.class, true)};
    }

    public void testLoginSuccessful()
    {
        String username = this.context.getParameter("username").asString();
        String password = this.context.getParameter("password").asString();

    }

For the test method testLoginSuccessful() an optional method paramTestLoginSuccessful() exists. The name of this method is derived by putting the prefix param in front of the capitalized method name of the unit. The return value is expected to be an array of the type ParameterDescription. The parameter description contains information about the type of the parameter and whether it is mandatory or optional.

If no such method is implemented the parameters are not validated during the validation run an may lead to problems during the test run if not provided correctly.

Test Run

During the test run the actual execution of the test description takes place. The execution of the test run succumbs to certain rules.

The execution starts at the root of the test hierarchy. If no errors occur, all tests and suites will be executed from top to bottom. Tests in parallel flows will be executed simultaneously. The flow is finished when all tests the flow have been executed.

In case of errors during the execution the reaction of the framework to these errors can be specified with so-called controls. Every flow in the test description can have several controls for the error types:

• severe error
• error
• severe failure
• failure
• success

Controls define which action shall be taken on which error. The following actions can be specified:

• next_test_noinits
• next_text_init
• next_flow
• stop

If the error type occurs during a flow the defined action is triggered.

next_test_noinits means that the next test in the flow is executed as if nothing has happened. The part noinits means that no inits are executed. Inits are the sum of setups and teardowns.

The action next_test_inits in case of an error makes the TestRunner first execute the teardowns and then the setups of the testsuite/flow before executing the next test thus providing an initial test environment.

next_flow ends the execution of the flow and only executes the teardowns of the current suite before continuing with the next flow.

stop means the whole test run will be terminated.

This influence on the execution control of the framework is only possible within flows. If errors happen during the setups of a suite the rest of the suite including flow and checkups are not executed because it makes little sense to execute tests after not being able to establish their preconditions. The checkups of tests with errors are not executed either. Teardowns on the other hand are always executed to keep the influence on following tests as low as possible. Errors during checkups and teardowns do not influence the execution control but only the test results.