Conformance Tester

Usage Examples Options

To execute tests using the test runner ct, you must write a test harness in C# that couples your implementation to ct. You can provide a test suite generated from a model program by otg, or ct can generate test cases on-the-fly from a model program as the test run executes. You can write a custom strategy in C# that ct uses to maximize coverage according to criteria you define.

An NModel test harness is called a stepper. To test a reactive (event-driven) system, you can write an asynchronous stepper that handles observable actions (events) in the implementation that are not under the tester's direct control.

Usage

ct /iut:<string> [/modelStepper:<string>] /reference:<string>+ [/mp:<string>]* 
[/coverage:<string>]* [/steps:<int>]* [/maxSteps:<int>]* [/runs:<int>]* 
[/observableAction:<string>]* [/cleanupAction:<string>]* 
[/internalAction:<string>]* [/waitAction:<string>]* [/timeoutAction:<string>]*
[/timeout:<int>]* [/continueOnFailure[+|-]]* [/logfile:<string>]* 
[/randomSeed:<int>]*  [/overwriteLog[+|-]]* [/testSuite:<string>]* 
[/fsm:<string>]* [/startTestAction:<string>]* <model>* @<file>

Examples

ct @ct_args.txt
ct /r:Stepper.dll /iut:ClientServerImpl.Stepper.Create /testSuite:ContractTest.txt
ct /r:ClientServer.dll ClientServer.Factory.Create /r:Stepper.dll ^
    /iut:ClientServerImpl.Stepper.Create /fsm:Scenario.txt /runs:1
ct /r:ClientServer.dll /mp:ClientServer /r:Stepper.dll ^
    /iut:ClientServerImpl.Stepper.Create /fsm:Scenario.txt /runs:1

Options

/?, /help
Displays usage information and exits.

/iut:<string>
Implementation under test, a fully qualified name of a factory method that returns an object that implements IStepper.

[/strategy:<string>]
A fully qualified name of creator method that takes arguments (ModelProgram modelProgram, string[] coverage) and returns an object that implements IStrategy. If left unspecified the default model stepper is used that ignores coverage point names (if any). (If a testSuite is provided, this option is ignored.)

/reference:<string>+
Referenced assemblies. (Short form: /r)

[/mp:<string>]*
Model programs given in the form M or M[F1,...,Fn] where M is a model program name (namespace) and each Fi is a feature in M. Multiple model programs are composed into a product. No factory method for the model program is needed if this option is used.

[/coverage:<string>]*
Coverage point names used by model stepper. (If a testSuite is provided, this option is ignored.)

[/steps:<int>]*
The desired number of steps that a single test run should have. After the number is reached, only cleanup tester actions are used and the test run continues until an accepting state is reached or the number of steps is MaxSteps (whichever occurs first). 0 implies no bound and a test case is executed until either a conformance failure occurs or no more actions are enabled. (If a testSuite is provided, this value is set to 0.) Default value: '0'

[/maxSteps:<int>]*
The maximum number of steps that a single test run can have. This value must be either 0, which means that there is no bound, or greater than or equal to steps. Default value: '0'

[/runs:<int>]*
The desired number of test runs. Testing stops when this number has been reached. Negative value or 0 implies no bound. (If a testSuite is provided, this value is set to the number of test cases in the test suite.) Default value: '0'

[/observableAction:<string>]*
Action symbols of actions controlled by the implementation. Other actions are controlled by the tester. (Short form: /o)

[/cleanupAction:<string>]*
Action symbols of actions that are used to end a test run during a cleanup phase. Other actions are omitted during a cleanup phase. (Short form: /c)

[/internalAction:<string>]*
Action symbols of tester actions that are not shared with the implementation and are not used for conformance evaluation. Other
tester actions are passed to the implementation stepper. (Short form: /i)

[/waitAction:<string>]*
A name of an action that is used to wait for observable actions in a state where no controllable actions are enabled. A wait action is controllable and internal and must take one integer argument that determines the time to wait in milliseconds during which an observable action is expected.

[/timeoutAction:<string>]*
A name of an action that happens when a wait action has been executed and no obsevable action occurred within the time limit provided in the wait action. A timeout action is observable and takes no arguments.

[/timeout:<int>]*
The amount of time in milliseconds within which a tester action must return when passed to the implementation stepper. Default value: '10000'

[/continueOnFailure[+|-]]*
Continue testing when a conformance failure occurs. Default value: '+'

[/logfile:<string>]*
Filename where test results are logged. The console is used if no logfile is provided.

[/randomSeed:<int>]*
A number used to calculate the starting value for the pseudo-random number sequence that is used by the global choice controller to select tester actions. If a negative number is specified, the absolute value is used. If left unspecified or if 0 is provided a random number is generated as the seed. Default value: '0' (Short form: /seed)

[/overwriteLog[+|-]]*
If true the log file is overwritten, otherwise the testresults are appended to the logfile Default value: '+'

[/testSuite:<string>]*
File name of a file containing a sequence of actions sequences to be used as the test suite.

[/fsm:<string>]*
File name of a file containing the term representation fsm.ToTerm() of an fsm (object of type FSM). Multiple FSMs are composed into a product.

[/startTestAction:<string>]*
Name of start action of a test case. This value is used only if a testSuite is provided. The default 'Test' action sybmol is considered as an internal test action symbol. If another action symbol is provided it is not considered as being internal by default. Default value: 'Test'

<model>*
Fully qualified names of factory methods returning an object that is a subclass of ModelProgram. Multiple models are composed into a product. No factory method arguments are needed if the /mp: option is used instead.

@<file>
Read response file for more options.