OaaTest Manual

last updated 11/21/2003

Resources

Contents

  1. Introduction
  2. Running OaaTest
  3. OTML File Format
  4. Automatically Generate OTML Files
  5. Index Files and OaaTest Input

Introduction

OaaTest is a java-based tool for testing Open Agent Architecture systems.  An OaaTest testcase initiates action in an OAA system by sending OAA events, then watches for specific events to confirm the expected behavior.  Testcases are written in an XML scripting language named OTML ("oatmeal").  OaaTest can unit-test specific OAA agents or system-test a collection of agents depending on the OTML file. It takes a directory tree of test files as input and runs the tests against a running OAA Facilitator and collection of agents.  OaaTest conforms to the JUnit API and uses JUnit tools to report test results, either the JUnit GUI test-result viewer or the ANT junitreport HTML generator. 

OaaTest is distributed as part of the core OAA distribution, download and unpack the most recent version from the OAA website.  All pathnames in this document are relative to the toplevel oaa2 directory of the distribution.

OaaTest is an integral part of the CALO Automated Test Framework and this document refers to it several times.  However, OaaTest does not depend on CALO, and non-CALO readers may safely disregard all such references.

requirements:
OaaTest runs on Windows and UNIX with a Java 2 VM version 1.4.1 or later.   It may work on Macintosh but has not been tested.  HTML reporting in OaaTest requires you have ANT installed.  GUI reporting does not require ANT.

limitations:
OaaTest cannot test direct_connect connections.  We suggest you disable direct_connect in your agents when using OaaTest.

Running OaaTest

The facilitator and all agents involved in a test suite must be running and ready before OaaTest runs the tests.  You may want to use the OAA StartIt application for agent startup/shutdown.  Run OaaTest after StartIt has done its work.  The CALO Automated Test Framework uses this method to fully automate starting/stopping agents and facilitator.

Sample Tests

OaaTest comes with sample tests to run on the sample agents shipped with the OAA 2.3 distribution.  To run the samples, start the facilitator and agents listed below.  This list is everything in the default StartIt configuration plus the Eliza agent.  Then use one of the methods in this section to run OaaTest  with the test file directory runtime/oaatest/samples/test2.
  1. DCG-NL Natural Language Parser Agent
  2. WebL: Weather Agent
  3. WebL: Employees Agent
  4. Eliza Agent

Methods of Running OaaTest

OaaTest conforms to the standard JUnit API for running tests.  This gives the flexibility to run OaaTest in different ways:

1) GUI test-reporting
  1. Edit runtime/scripts/oaatest_gui.bat (Windows) or runtime/scripts/oaatest_gui.sh (UNIX) and set TEST_IN to point to your test files,
  2. Run oaatest_gui.bat or oaatest_gui.sh.
  3. The JUnit test viewer appears and the progress bar turns solid green if tests succeed.
  4. The console gives the log of interactions between OaaTest and the facilitator.

2) HTML test-reporting (requires ANT)
  1. Your ANT installation must have junit.jar or junit-3.8.1.jar in its lib directory ($ANT_HOME/lib).  If its not there, copy lib/jar/junit-3.8.1.jar from your OAA distribution.
  2. Edit runtime/scripts/oaatest_html.xml:
    1. set oaatest.in to point to your test files,
    2. set oaatest.out to the desired HTML output location,
  3. From runtime/scripts, run "ant -f oaatest_html.xml".
  4. See the log of interactions between OaaTest and the facilitator.
  5. View the HTML report in ${oaatest.out}/html/results.html.

3)Part of CALO Automated Test Framework

4) Other JUnit Tool
You can run OaaTest using any tool that runs JUnit tests.  For example, the Eclipse development environment can run OaaTest.
  1. Add these jar files from lib/jar to the tool's classpath:
    1. oaa2.jar
    2. antlr-oaa.jar
    3. concurrent-1.3.1.jar
    4. log4j-1.2.7.jar
    5. junit-3.8.1.jar
  2. Add runtime/oaatest/jar/oaa2oaatest.jar to the tool's classpath.
  3. Set the oaatest.in Java system property to point to your test files, perhaps through the Java VM command line, "-Doaatest.in=c:\my\test\files".
  4. Set the JUnit test class to be com.sri.oaa2.tools.oaatest.OaaTest

OTML File Format

The OaaTest Markup Language (OTML or "oatmeal") is an XML script language for writing OAA test cases.  There is one top-level <testcase> element per file.  It contains <send>, <toFac>, <fromFac>, <parallel>, and <prompt> elements.  Use <send> to initiate some behavior in an OAA agent or the system as a whole.  Then use <toFac> and <fromFac> to check for the expected response, indicated by events to and from the facilitator.  Use <parallel> when you expect several events, but the relative order is undefined.  Use <prompt> for user input.

Example:

<testcase>
    <send>
        oaa_Solve(weather([in(houston)],_),[blocking(false)])
    </send>
    <toFac>
        ev_solved(_,_,_,
                  weather([in(houston)],_),
                  _,
                  [weather([in(houston)],HoustonWeather)])
        <regex
            var="HoustonWeather"
            value="'The weather in houston is currently .* temperature .*'"
        />
    </toFac>
</testcase>

The example testcase sends a request for the weather in Houston, then waits for some agent to send a matching ev_sovled() event to the facilitator.  The event has a regular-expression constraint so the variable "HoustonWeather" matches strings like 'The weather in houston is currently partly cloudy with a temperature of 102 degrees.'.

OTML Syntax

<testcase>

The toplevel XML element.

contents:
List of <send>, <toFac>, <fromFac>, <parallel>, and <prompt> elements.

attributes:
timeout (integer, optional):  Set the default timeout value (seconds) for <toFac>, <fromFac>, and <parallel> tags.  5 seconds if not specified.

<send>

Send a single OAA event to the facilitator.

contents:
Contains a fully specified event as an ICL term or the following syntax (see note):
  oaa_Solve(Goal,Params)
  oaa_AddData(Data,Params)
  oaa_RemoveData(Data,Params)
  oaa_ReplaceData(Data,Params)
  oaa_LoadData(Data,Params)
  oaa_SaveData(Data,Params)
  oaa_AddTrigger(Type,Condition,Action,Params)
  oaa_RemoveTrigger(Type,Condition,Action,Params)

attributes:
none

<toFac>/<fromFac>

Match a single OAA event sent to/from the facilitator.

contents:
An ICL term (see note) followed by zero or more Constraint tags.  The ICL term may contain named  Prolog variables (words beginning w/ a capital letter or underscore) or the unnamed variable (underscore).  Each constraint acts on one of the named variables in the ICL term.  An event to/from the facilitator matches if it unifies with the ICL term and satisfies all constraints.

attributes:
timeout (integer, optional): Specify in seconds how long to wait for the event.  Default is 5 seconds or value from <testcase> tag if set.

Constraints on <toFac>/<fromFac>

Each constraint tag is of the form <name var="Var" value="Val" /> where name is one of the tags given below, Var is a variable from the preceding ICL term, and Val is interpreted as below.

Name
 Type for "value" Parameter
 Meaning
gt
 number
 Var will only match int or float numbers greater than Val, e.g. <gt var="X" value="3"> means X > 3.
lt, le, ge
 number
 similar to gt  for <, <=, and >=.
regex
 regular expression
 Var will only match ICL terms that match the regular expression given in Val.  The regular expression syntax is that of  the java.util.Regex.Pattern class. When using '|' for OR, add an extra set of parenthesis around the entire OR expression, e.g. <regex var="FirstName" value="(('Joe')|('Mary')|('Elmo')|('Alexa'))" />
contains
 ICL term
 Var will only match ICL lists that contain the ICL term Val as an element.
Match list L iff some element of L unifies with Val.
member
 ICL list
 Var will only match ICL terms that are a member of the ICL list Val.
Match term T iff T unifies with some member of Val.
subset
 ICL list
 Var will only match ICL lists that are a subset of the ICL list Val.  Duplicates are ignored in both Val and potential matching lists.
Match list L iff all elements of L unify with some element of Val.
superset
 ICL list
 Var will only match ICL lists that are a superset of the ICL list Val. Duplicates ignored.
Match list L iff all elements of Val unify with some element of L.
equalset
 ICL list
 Var will only match ICL lists that contain the same terms as the ICL list Val.  Order and duplicates are ignored.
Match list L iff all elements of L unify with some element of Val  AND
all elements of Val unify with some element of L.

Note: set operations use unification to decide equality.  For example, [dog(rover),cat(X)] is a subset of [iguana(earl),cat(whiskers),dog(Y)].

<parallel>

Match several events sent to/from the facilitator, when the order of events is not known.  Especially useful if a <send> event causes multiple agents to act.  You might require certain events from multiple agents, but don't know which will complete first.

contents:
Multiple <toFac> or <fromFac> tags giving the list of events to match.  The <parallel> tag will succeed when it has seen a matching event for each of its <toFac>/<fromFac> children.  Order of the children does not matter.

attributes:
timeout (integer, optional): Minimum time in seconds to wait for all events to complete.  Actual timeout will be maximum of this value and all children's timeouts.

<prompt>

Prompt the user with a message dialog.  Block execution of OTML script until user presses OK. Useful for testing user-interface components, e.g. <prompt message="Enter 'list all addresses' in the window and press OK"/>.  May not be used inside <parallel>.

contents:
Text to display in dialog.  Use this or message attribute, but not both.

attributes:
message (string, optional): Text to display in dialog.  Can only be used if text contents are empty.

ICL terms in OTML files: 

Since OTML is an XML language, ICL terms must be valid XML.  In particular, any occurrences of < and & must be escaped as &lt; and &amp;  A single quote ' inside an ICL string is escaped with two quotes '' as usual in ICL.

<testcase>
    <send>
        oaa_Solve(foo('It''s true that 3 &lt; 4, but not 3 &gt; 4'),[])
    </send>
    <send>
        oaa_Solve(foo('This is an ampersand &amp;'),[])
    </send>
</testcase>

It is ok to split ICL terms over multiple lines, but be careful splitting quoted strings:

<send>
    oaa_Solve(foo(a,b,c,d
                  e,f,g,h,
                  'I am a quoted string with a newline right here.
I am the same quoted string after the newline.'),
              []
             )
</send>

Hints for Writing OTML Files

Make your tests general to all hardware/agent configurations that might run the tests.  In particular don't include any IP addresses, machine names, or agent ID numbers.  Replace them with '_'.

Write many small tests, each testing a single aspect of your system. Long scripts make it difficult to pinpoint failures.

To make a unit test for a single agent, start with a <send> item to run one of the agent's solvables.  Then add <toFac> items to look for the agent's response to the facilitator or <fromFac> items to look for the facilitator forwarding the response back to OaaTest.

To make a system test for a collection of agents, use <send> items to mimic user-interface agents or whatever starts the system running.  Add <toFac> and <fromFac> items to look for critical events in the system interaction.  When there is ambiguity about the ordering of events, use a <parallel> item with nested <toFac> and <fromFac> items.  You might start by using the Monitor Agent to observe the system and save the Monitor log as an OTML file, see next section.

Automatically Generate OTML Files

As of OAA 2.3, the OAA Monitor Agent can generate OTML files.  Given a collection of agents, you can make a quick regression test by recording event interactions during some activity then saving the sequence of events as an OTML file.

The Monitor now has an new menu item "Save Log as OTML" under the "Event log" menu.  First, record a log of interactions among OAA agents, then "Save Log as OTML".  You will have the choice of generating an "active" or "passive" OTML script.  An active OTML script uses <send> events to initiate behavior in an OAA system, then uses <toFac> and <fromFac> to match events and confirm the expected behavior.  A passive OTML script has no <send> events.  Some application other than OaaTest (perhaps a GUI) initiates behavior, then the OTML script uses <toFac> and <fromFac> to confirm the expected behavior.  To generate an active script, you must select an "Initiating Agent" from the list of agents involved in the logged events.  The generated script will actively <send> whenever the Initiating Agent sent an event.  For example, if you used the Debug Agent to initiate behavior while recording events, then the Debug Agent is your Initiating Agent.

Except for trivial cases, the generated OTML file will not be exactly what you want.  Use the generated file as a template and edit it to make the test you want.  Remove events not relevant to your test.  Make <toFac>/<fromFac> event patterns more abstract, replacing overly-specific ICL terms with "_".

Example:

Generate an OTML file from the sample OAA agents.
  1. Run the Facilitator and these agents from the OAA distribution (all in the default StartIt configuration):
    1. DCG-NL
    2. WebL: Employees
    3. Java Debug Interface
  2. Run the Monitor agent.
  3. The Monitor connects and you see the agent icons in the Monitor window.
  4. In the Monitor window, click the record icon (yellow tape cassette) to turn on event recording.
  5. In the Debug Interface window in the "English Request" box, type "Who is adam cheyer's boss" and click the send button (unlabeled button to the right of  the text box).
  6. The result "Ray Perrault" appears in the Debug window.
  7. In the Monitor window, click "Show Log".  You should see the events involved in the previous action, about eight of them.
  8. In the Monitor window, select the menu item "Event Log|Save Log as OTML".
  9. Choose oaa_debug_agent as the Initiating Agent.
  10. Save the file somewhere as manager.otml.
  11. Exit the Monitor and Debug agents.

Run the OTML file to test the behavior recorded above.
  1. Make sure the Facilitator, DCG-NL, and WebL:Employees agents are running.
  2. Run OaaTest on manager.otml using GUI test-reporting, see Methods of Running OaaTest.  TEST_IN in runtime/scripts/oaatest_gui.bat will point to your manager.otml file.
  3. The progress bar in the JUnit test-viewer turns green and the test passes.

Index Files and OaaTest Input

OaaTest takes a single filename as input and creates the list of OTML files to run.  It traverses the directory tree and adds all *.otml files or follows the instructions in any Index Files.

If the Input file:
is a directory
Look for index.idx and follow instructions in that index file.  Otherwise, add all .otml files and traverse all subdirectories.
ends in ".otml"
Add the OTML file to the list.
ends in ".idx"
Follow instructions in the index file.
Most OaaTest tests can be implemented with a single OTML file, run the instructions in order and halt on the first failure.  But, some tests require greater complexity: guaranteed setup and teardown of test resources, or repeating some action multiple times.  These tests are implemented with several OTML files and an Index file for control.  Index files allow you to:
  1. Control the order OTML tests in a directory are run.
  2. Repeat an OTML test or collection of tests multiple times.
  3. Run only a subset of the OTML files in a directory, e.g. you could have quick.idx run a quick subset of tests and complete.idx run the entire time-consuming set.
Example Index File:

<index>
    <testcase file="setup.otml"/>
    <repeat count="2000">
        <testcase file="set.otml"/>
        <testcase file="get.otml"/>
    </repeat>
    <testcase file="teardown.otml"/>
</index>

This example first runs the setup.otml test file, then runs set.otml followed by get.otml 2000 times, and completes by running teardown.otml.  setup.otml prepares the OAA system for testing, set/get.otml are the body of the test, and teardown.otml cleans up.  If  the test fails during set/get, the repeat stops but teardown still cleans up properly.

Index File Syntax

<index>

The toplevel XML element.  Run all child elements in order.  If a child test fails, continue with the next element.

contents:
List of <testcase>, <dir>, and <repeat> elements.

attributes:
none

<testcase>

Run a single OTML test file.

contents:
none

attributes:
file (required): relative filename of the OTML file.

<dir>

Traverse a subdirectory.  If it contains index.idx, follow the instructions there.  Otherwise, add all *.otml files and traverse all sub-subdirectories.

contents:
none.

attributes:
dir (required): relative pathname of the subdirectory.

<repeat>

Repeat the contents of this tag multiple times.  If any test inside the repeat fails, finish the current round but do not repeat again.  <repeat> elements may be nested, but there is rarely a reason to do so.

contents:
List of <testcase>, <repeat>, and <dir> elements.

attributes:
count (integer, required): number of times to repeat.

TO DO

Improved HTML Reporting

The HTML reports generated by ANT's junitreport are not especially well-suited to OaaTest output.  We plan to make HTML reports simpler and more legible.  Reports will include links to the OTML files, line numbers where tests failed, and a division into top-level summary page and individual detail pages.

Copyright © 2003 by SRI International