Adept Client Application Guide


Table of Contents



Overview

This guide describes how to enable software applications for use with the Adept task learning system. In short, an Adept-enabled application must provide instrumentation to log steps that are performed by the user within the application along with corresponding automation for subsequently recreating those steps when learned procedures are executed. The current APIs in Adept support interaction with Java software only.

Adept supports interactions with both individual client applications and suites of complementary client applications, with the latter resulting in the learning of multi-application procedures.

Action Model

The key requirement for enabling a client application to work with Adept is the creation of an action model. The action model specifies the application's operations in a form suitable for task learning. Each action in the action model has a name and parameters (input and output), and represents a single conceptual operation within the application.

The action model that an application presents to Adept fulfills several important roles. 1) The elements of the action model constitute the building blocks for task learning. As such, the design of the action model significantly affects learning and generalization. 2) Actions in the action model are executed from learned procedures; hence the action model serves as an API between Adept and the application. 3) The action model specifies the level at which the user views and manipulates learned procedures. As such, the action model should be defined in terms to match the users' conceptualization of their interactions with the application.

A sequence of actions is modeled as a dataflow, where the effects of executing an action are characterized by its inputs and outputs. In a strict dataflow model, input and output parameters must be distinct (i.e., a parameter cannot be both an input and an output) and all relevant state changes resulting from executing an action must be captured by the action outputs.

A core assumption in the dataflow model is that data is immutable: it is produced (output) and consumed (input), but it is not changed. This is similar to the notion of pipes in Unix, where the output of one command is passed in as input to the next. Adept is thus best-suited to applications where actions involve data passing between actions—for example, scientific workflows that process and analyze data. It is not designed for applications better captured through a state-based model, where actions primarily involve changes to the state of objects or the environment. For example, a mobile robot whose actions include carrying objects and moving them between rooms would be natural to model in terms of state changes to its current location and to what it is carrying. Modeling such actions using pure dataflow is problematic because there is no way to specify a state change; only input and output of data objects may be expressed.

Creating Learned Procedures

Adept is a programming by demonstration or learning by demonstration system. In systems of this type, applications provide instrumentation that notifies Adept of executed user actions. Thus, Adept learns from user actions as the user normally operates applications, with the only added requirement being that the user indicate the start and end of the demonstration. The resulting sequence of actions is called the demonstration.

The action model specifies the actions for capture as well as their parameters. These actions are collected and passed to the Adept learning component, which generalizes them into a parameterized procedure that may be subsequently executed, with the same or with different parameters. Capturing and delivering these instrumentation notifications is one principal task involved in enabling Adept for a client.

Executing Learned Procedures

Adept provides a user interface for executing learned procedures. This invokes Adept's execution component on the procedure, which sends automation requests to the application to execute the individual actions contained within the procedure.

The client application services these requests by executing its normal semantics for that action, and then returning the values for any output parameters of that action. Listening for and responding to automation requests is the other principal task involved in enabling Adept for a client.

Architecture

Adept Task Learning Architecture

LAPDOG is the task learning component of Adept. LAPDOG creates learned procedures from actions demonstrated by the end user and collected by instrumentation in the client applications.

Lumen is the execution component of Adept. It executes learned procedures on demand from the end user.

The Procedure Library is a data store on the client file system for persisting learned procedures. It resides in the platform-specific location that is conventional for storing application data.

The JMS Spine Server is a message-passing service that connects all Adept components. It utilizes Java Messaging Service (JMS), providing a server that conveys all message traffic in the system.

The Bridge is an API layer that provides access to Adept functionality for learning, executing, and visualizing procedures. It also insulates client applications from the details of JMS message passing.

Adept Control is invoked by the UI to start several Adept components and provides some core facilities used by Adept applications.

The Adept UI provides the user interface for Adept. The UI supports control of learning sessions, initiation of the execution of learned procedures, and management of procedures in the Procedure Library.

The Procedure Editor allows Adept users to view, edit and create procedures.

Image Loader and Novo are simple Java applications that are Adept enabled, providing a sample application illustrating instrumentation and automation for an Adept client, and operation of the Adept UI as shown in the User Guide.


Task Learning

Task learning—the creation of procedures that perform user tasks—is done by Adept's LAPDOG module. LAPDOG performs task learning by converting a demonstration trace into an executable, parameterized procedure that reproduces the demonstrated activity and generalizes it so that the demonstrated task may be executed in the future to perform a range of similar tasks. This section discusses some considerations for the effective use of LAPDOG for Adept applications.

The specifics of the action model significantly affect the generalizations that are performed. To discuss this, we use a simplified notation for actions in the action model, suppressing some implementation details. In demonstration examples, to highlight dataflow, input arguments are preceded by a "+"; while a "-" precedes output arguments. Similarly, in procedure examples, input and output parameters are designated by "+" and "-", respectively. Variables in procedures, including parameter variables, are designated with a leading dollar sign. Argument types are omitted unless typing is significant to the example, in which case they are notated as ":TypeName" after the action argument. The procedure resulting from learning is notated as

ProcedureName(parameters) {
   action1(arguments)
   action2(arguments)
   ...
   actionN(arguments)
}
Action models specify the action name as well as the name and mode (i.e., input or output) of each parameter. For example:
Convert(+Infile, +Format, -Outfile)
Delete(+File)
GetCreationDate(+File, -Date)
List(+Directory, -FileList) 

Parameter Generalization

It is possible to learn parameterless procedures (i.e., macros) that perform the exact same sequence of operations every time they are executed. Consider a one-action demonstration:
deleteFile(+"MyFile.txt")
From this demonstration, Adept can learn a parameterless procedure that always deletes the file "MyFile.txt". However, the true power of Adept is to learn procedures with parameters, such that the procedure can be applied to a range of tasks similar to the demonstrated task, as opposed to only that specific, verbatim task. The process by which parameters are created and associated with demonstrated values is called parameter generalization.

For the action model

Convert(+Infile, +Format, -Outfile)
Delete(+File)
GetCreationDate(+File, -Date)
List(+Directory, -FileList) 
consider the following execution trace for a demonstration within a file system:
Convert(+manual.pdf, +"HTML", -manual.html)
GetCreationDate(+manual.html, -2009-03-23)
LAPDOG generalizes a demonstration by converting the arguments in the demonstration to variables in the learned procedure. Further, LAPDOG analyzes the dataflow in the demonstration, looking for support relationships—cases in which an output of an action is used as an input in one or more subsequent actions. In a support relationship, the supporting argument and all supported arguments are replaced or co-designated with a common variable in the learned procedure:
Convert($Infile, "HTML", $File)
GetCreationDate($File, $Date)
The effect of this co-designation is to force all support-related inputs and outputs to have the same value when the learned procedure is executed.

This example illustrates the basic generalization performed via learning by demonstration. Values in the sequence of demonstrated actions are replaced with variables; this is termed variablization. If no support is found for an argument, then this is made an input parameter to the learned procedure, obliging the user to provide it upon execution. This process of support analysis and variablization is termed parameter generalization.

Limiting variablization by designating action model parameters as ungeneralizable is possible. Such parameters are never variablized but rather remain as constants in the learned procedure. For example, if the Format parameter of the Convert action is made ungeneralizable, learned procedures containing it will always convert to the demonstrated format ("HTML" in this example). Additionally, certain constants—the null value, the empty string, empty list, and empty set—are never generalized. All other values are variablized.

The action model affects the expressive power of learned procedures. For example, an alternative formulation of the action model might model file deletion as

deleteFile(+"MyFile" +"txt")
which allows learning a variety of procedures with various parameterizations, controllable from the Adept UI. For example, a two-parameter version looks like
DeleteFile(+$basename +$extension) {
   deleteFile($basename $extension)
}
If the underlying system allows wildcards, one could execute
DeleteFile("*" "txt")
to delete all text files in some unspecified context.

In some cases, learning will create an unintended input parameter. For example, a demonstration that increments a number by one:

add(+23 +1 -24)
might generalize to
AddTwoNumbers(+$number1 +$number2 -$result) {
   add($number1 $number2 $result)
}
when the actual intent is to always increment a number by one. As an alternative, the application could make a more specific action available:
addConstant(+$number +$constant -$result)
in which $constant is designated as an ungeneralizable parameter. This forces the corresponding value in the demonstration to be retained as a constant:
IncrementNumber(+$number1 -$result) {
   add($number1 1 $result)
}
The tradeoff is that the application must make this additional primitive available, and the user must know to use it. Such tradeoffs are typical of action model design considerations.

Support Relationships

During parameter generalization, LAPDOG determines which values support other values. For a learned procedure to be valid, all inputs to all its actions must be supported. For an action, an input value is supported if the value is available to the learned procedure when that action is executed, either as input to the procedure, as output by a previous action, or as a known function (discussed in detail below) of supported values.

Support determination depends on the types of the values involved.

Support relationships for compound values are defined recursively, permitting an element of an arbitrarily structured data value to support another if their types are compatible.

For example, given a demonstration of some string-manipulation actions:

concatenate(+"the quick " +"brown fox" -"the quick brown fox")
removeBlanks(+"the quick brown fox" -"thequickbrownfox")
uppercase(+"the quick brown fox" -"THE QUICK BROWN FOX")
the learned procedure is
P(+$string1 +$string2 -$out1 -$out2 -$out3) {
   concatenate($string1 $string2 $out1)
   removeBlanks($out1 $out2)
   uppercase($out1 $out3)      
}
Because the first occurrence of "the quick brown fox" supports values in the second and third actions, they are all co-designated with the same variable. Further, because the first occurrence is computed, it need not, and should not, be an input parameter.

Ambiguous Supports

Ambiguity occurs when two or more supports are available for a given value. In the demonstration
findZipCode(+"alice" -"12345")
findZipCode(+"bob" -"12345")
printZip(+"12345")
the support for the "12345" in printZip is ambiguous. LAPDOG resolves such ambiguities during generalization by using the most recent supporting value:
findZipCode($name1 $zip1)
findZipCode($name2 $zip2)
printZip($zip2)
Using distinct data types can help prevent unwanted generalizations from arising from ambiguous supports. For example, the demonstration (with relevant argument types made explicit):
findZipCode(+"bob" -"12345":Zip)
findEmployeeId(+"bob" -"12345":Id)
printZip(+"12345":Zip)
contains two potential supports for "12345" in the last action. However, it generalizes uniquely to
findZipCode($name $zip)
findEmployeeId($name $id)
printZip($zip)
due to the types of the action parameters.

Collections, Structures and Functions

The action model may define arguments as collections of values that are all of the same type. For example, we can define an action that finds the names of all employees of a company:
getAllEmployeeNames(-["alice" "bob" "carl"])
Collections may be designated as lists, sets, or bags.

The action model may define arguments as structures, which are arbitrarily structured data similar to records or structures in programming languages. For example, we can define a structure to represent an employee, containing their first name, last name, and employee ID:

findEmployee(+12345 -<firstName="Alice" lastName="Ames" id=12345>)
The elements of a structure are named. Each element may be of any type, including lists, sets, and other structures. Supports and variablization are computed both for the structure itself, and for each element. Like any other value, one structure can support another in its entirety:
findEmployee(+12345 -<firstName="Alice" lastName="Ames" id=12345>)
printEmployee(+<firstName="Alice" lastName="Ames" id=12345>)
generalizes to:
findEmployee($id $employee)
printEmployee($employee)
Structure elements can support other values. The following demonstration of constructing a full name from first and last names:
findEmployee(+12345 -<firstName="Alice" lastName="Ames" id=12345>)
concatenate(+"Ames" +"," -"Ames,")
concatenate(+"Ames," +"Alice" -"Ames,Alice")
generalizes to:
findEmployee($id $employee)
concatenate((mapGet $employee "firstName") "," $string1)
concatenate($string1 (mapGet $employee "lastName") "Ames,Alice")
Note the generalization of the argument "Ames" to the function (mapGet $employee "firstName"). This means that the element named "firstName" of the structure associated with $employee is used as the argument of the action. The functional notation is internal to Adept procedures; in the Adept UI, the mapGet appears as 'Field firstName in $employee'.

Besides structures, the accessor functions first($List) and last($List) are available for lists. Thus, a demonstration

getAllEmployeeNames(-["alice" "bob" "carl"]: list<string>)
findZipCode(+"alice" -"12345")
printZip(+"12345")
findZipCode(+"carl" -"67890")
printZip(+"67890")
generalizes to:
getAllEmployeeNames($people)
findZipCode(first($people) $zip1)
printZip($zip1)
findZipCode(last($people) $zip2)
printZip($zip2)
If a list has only one element, both first($List) and last($List) are valid generalizations for element references. For example, consider an organization in which an employee typically has one supervisor, but in unusual cases could have several. The demonstration
findSupervisorNames(+"bob" -["dan"] )
findEmail(+"dan" -"dan@xyz.com")
could generalize to either
findSupervisorNames($employee $supervisorList)
findEmail(first($supervisorList) $email)
or
findSupervisorNames($employee $supervisorList)
findEmail(last($supervisorList) $email)
with equal validity. Instead of arbitrarily choosing one, in the case of such singleton lists, a third generalization is preferred by default:
findSupervisorNames($employee $supervisorList)
findEmail(only($supervisorList) $email)
The only function means: use the sole element of the list; it causes an error if the list does not contain exactly one element. List types allow a generalization preference to be specified as to whether to use only, first, or last in the case of singleton lists. For other collections (sets and bags), only is always used.

In addition to accessor functions, constructor functions exist, which support a list, set, or structure by constructing it from individually supported parts. For example, the demonstration

inputEmployeeName(-"Bob" -"Baker")
generateNewEmployeeId(-67890)
createNewEmployee(+<firstName="Bob" lastName="Baker" id=67890>)
generalizes to
inputEmployeeName($firstName $lastName)
generateNewEmployeeId($id)
createNewEmployee((mapGen "firstName" $firstName "lastName" $lastName "id" $id))
with mapGen being the Adept-internal name for the structure-constructor function; in the Adept UI it is shown as a structured value rather than a function invocation. Structures obey strict typing with regard to support determination. One structure value does not support another unless they are of identical types, even if the names and the values of their components exactly match.

Structures and collections may be partially supported. For example, in the demostration:

inputEmployeeName(-"Bob" -"Baker")
createNewEmployee(+<firstName="Bob" lastName="Baker" id=67890>)
the fields "Bob" and "Baker" are supported, but the field 67890 is not. LAPDOG by default treats the structure or collection as unsupported; in this example, the structure argument in whole would be generalized to a procedure parameter, ignoring the partial supports:
P(+$employee) {
    inputEmployeeName($firstName $lastName)
    createNewEmployee($employee)
}
However, a structure or collection type may specify a generalization preference permitting generalization by construction from partial supports. This permits the generalization to:
P(+$id) {
    inputEmployeeName($firstName $lastName)
    createNewEmployee((mapGen "firstName" $firstName "lastName" $lastName "id" $id))
}
The resulting procedure takes only the unsupported field of the structure as an input, rather than the entire structure, and constructs the structure from it and the supported fields. The user of the procedure may now enter one field value instead of three when running the procedure.

Using the generalize-by-construction preference can lead to many input parameters for the procedure. This is particularly true for collections, which may have an unlimited number of elements. For this reason, this preference allows an optional limit on the number of inputs allowed (default is to allow unlimited inputs); if this limit is exceeded, the structure or collection in whole is made a procedure parameter instead.

Loop Learning

If the action model contains lists or sets, Adept may learn loop generalizations. For example, consider the demonstration:
getAllEmployeeNames(-["alice" "bob" "carl"]:list)
findZipCode(+"alice" -"12345")
printZip(+"12345")
findZipCode(+"bob" -"12345")
printZip(+"12345")
findZipCode(+"carl" -"67890")
printZip(+"67890")
Here, list values are delimited with [ ], and the list type is made explicit. The demonstration contains a pattern of repetitions of the findZipCode and printZip actions. The first argument to findZipCode refers in turn to each element of the list supported by the first action in the demonstration. Moreover, the support relationships within the repeated actions are consistent--the output of findZipCode is used consistently as the input to the subsequent printZip. This action pattern forms a valid iteration over a list and generalizes to a procedure containing a loop:
getAllEmployeeNames($people)
for $person in $people do
   findZipCode($person $zip)
   printZip($zip)
od
LAPDOG is constrained to finding loops over collections (lists, sets, bags) that are explicit within the demonstration. In the case of lists, the elements of the list be iterated over in order, and the sequence of repeated actions (the loop body) must be identical except for the references to (including functions over) those elements.

LAPDOG can detect parallel iteration over multiple lists if the lists are of the same length and their elements are accessed consistently within the loop body. Loops may also generate list values that support subsequent lists. The outputs of a repeated action that form a loop body form a list that can be used to support list inputs later in the demonstration. Here is a richer example of loop learning, using abstract actions:

A(-[1 2 3])
B(-[a b c])
C(+a +1 -a1) 
C(+b +2 -b2)
C(+c +3 -c3)
D(+[a1 b2 c3])
This generalizes to
A(-X)
B(-Y)
for U in X, V in Y building Z do
   C(+U +V -W)
   W accumulate Z
od
D(+Z)

Dataflow Completion

Ideally, instrumentation will capture all of the user's actions. However, certain actions may be awkward to demonstrate (for example, indicating that an email address is that of the host of a meeting) or they may be mental actions with no associated physical gesture (for example, choosing to add one's initials to a document name). This leads to an implicit dataflow, where inputs to succeeding actions are not directly supported by outputs of preceding actions. For example, a demonstration where the user appends the current date to the name of a file:
selectFile(-"Report")
renameFile(+"Report" +"Report20101215")
would be generalized to a procedure requiring both the original and new names to be input:
RenameFile(+$oldfile +$newfile) {
   selectFile($oldfile)
   renameFile($oldfile $newfile)
}
However, given an appropriate library of information-producing actions, LAPDOG can infer the actions necessary to complete the dataflow. For example, given actions to retrieve today's date and to concatenate strings, the demonstration can be extended to:
selectFile(-"Report")
todaysDate(-"20101215")
appendString(+"Report" +"20101215" -"Report20101215">
renameFile(+"Report" +"Report20101215")
and a more general procedure that captures the rationale behind the new name can be learned:
RenameFile(+$oldfile -$newfile) {
   selectFile($oldfile)
   todaysDate($date)
   appendString($oldfile $date $newfile)
   renameFile($oldfile $newfile)
}
There are two categories of information-producing actions: completers and supporters. Completer actions are just like regular actions except that they have no instrumentation counterpart: they cannot be demonstrated and can only be inferred. Supporter actions are both demonstrable (and hence must be instrumented) and inferrable for dataflow completion. Information-producing actions are considered by LAPDOG as part of the parameter generalization process as it searches for a maximally general procedure.

There is another category of actions that can be used to infer implicit dataflow: context actions. They are not demonstrated, nor are they added by LAPDOG, but may be proactively inserted by the application into the demonstration for possible use in dataflow completion. Such actions are intended to capture properties of the environment within which the demonstration is performed. For example, context actions might be used to provide information such as the current user, the user's home directory, the operating system version, and the current date and time. If the context information is used to support actions in the demonstration, the learned procedure retains the corresponding context actions. Otherwise, if an inserted context action does not support any dataflow, it may be removed by Adept from the learned procedure (although Adept currently does not do so) or by the user through the Editor, without affecting the procedure's behavior.

Information-producing actions and context actions serve to capture more dataflow connections between the actions of the demonstration, providing greater generalization and resulting in procedures with fewer input parameters. However, dataflow completion can be an expensive search process, so care must be taken to define only information-producing actions and context actions that are useful and to ensure their proper application through strong parameter typing.


Creating the Action Model

The action model for a client application fulfills several important roles:

  1. The elements of the action model constitute the building blocks for task learning: as such, the design of the action model significantly affects learning and generalization.
  2. The actions in the action model are instrumented and automated, serving as an API between Adept and the application.
  3. The action model characterizes the level at which the user views and manipulates learned procedures.

The action model is expressed as an XML file. For full examples, see the Image Loader action model file and the Novo action model file.

This section discusses the various semantic options for expressing actions and the data types of their arguments, as well as how to define them in the action model XML file. This is followed by design guidelines in formulating an action model for an application.

The Action Model File

An action model file has the following structure:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="./am-to-html.xslt"?>
<actionModel version="2.1.a">

<!-- Types and Actions -->

</actionModel>

The action model file contains specifications for types, actions, and action parameters. The action model contains a required version designator on the actionModel tag. This assists the application in action model maintenance. It is also used internally by Adept, as a part of every action and type name. This allows Adept to detect and prevent execution of procedures using any action model version other than the one currently registered.

In addition to the various element-specific tags (described in the following subsections), the action model may contain various metadata tags of the form:

<metadata key="foo" value="bar"/>
Metadata enables markup of the action model to improve user understanding of the actions and types displayed in the user interface. Keys and values for each action model element are described in the following subsections. Any metadata whose key is not listed in this document is silently ignored. Metadata is optional; that is, leaving out metadata will not adversely impact Adept's functionality, only the UI display.

The stylesheet am-to-html.xslt is provided to enhance action model readability. The action model of an application may be split into several XML files by using the require tag. For example:

  ...
  <require url="./common-types.xml" />
  ...

Data Types

All action parameters have a data type. The typed approach to data representation is done for a number of reasons: 1) the semantics of actions are clearer, 2) strong typing can greatly enhance learning by constraining generalization and, in some cases, dramatically pruning the space of potential procedures, 3) the API is clearer, and 4) input values are more natural to enter when executing a learned procedure. There are several classes of data types supported in Adept:

Built-in Primitive Types

Adept provides several primitive data types that can be referenced immediately in an action model:

The following example, taken from the Novo action model, illustrates referencing a primitive type in an action declaration:

    <action id="dispenseShapes">
        <inputParam id="numShapes">
            <typeRef typeId="integer"/>
        </inputParam>
        ...
    </action>

Note that if a primitive type is used by multiple action parameters that have different intended semantics, Adept will not know about the semantic differences between the usages of the type and may generalize values that are semantically unrelated. For example, if an action parameter is declared that represents a person's age and uses the primitive type integer, another parameter intended to represent the number of days in a month also declared as an integer may result in an incorrect generalization if a value such as "30" is provided for both parameter values, regardless of whether the parameters are declared in the same action. In order to prevent this, it may be necessary to use "application types" where semantic differences exist.

Application Types

Application types are used to define custom semantic types in a client application. Generally an application type is built on a simple primitive type from Java, but any class fulfilling the following requirements may be used:

Adept provides the class com.sri.pal.CustomTypeFactory and a default implementation ToStringFactory to manage converting between string and object representations of custom types, though no examples of their usage are provided.

The following type declarations declare application types built on java.lang.Integer and java.lang.String respectively. These types are semantically separate from the built-in integer and string primitive types. Note that application types cannot be declared with the same name as a built-in primitive type, such as integer.

    <type id="myInteger">
        <description>An integer value</description>
        <custom>
            <javaType>java.lang.Integer</javaType>
        </custom>
    </type>
 
    <type id="myString">
        <description>A string value</description>
        <custom>
            <javaType>java.lang.String</javaType>
        </custom>
    </type>
 

Several application types that reference the same javaType may be specified. For example, an application using postal codes may not want to confuse those with other strings:

    <type id="ZipCode">
        <description>A U.S. postal code</description>
        <custom>
            <javaType>java.lang.String</javaType>
        </custom>
    </type>
 

Generally, the values of distinct types do not support each other. An exception is that a hierarchy of primitive types may be specified:

    <type id="ZipCodeShort">
        <description>A 5-digit U.S. postal code</description>
	<inherit parent="ZipCode"/>
        <custom>
            <javaType>java.lang.String</javaType>
        </custom>
    </type>

    <type id="ZipCodeLong">
        <description>A 9-digit U.S. postal code</description>
	<inherit parent="ZipCode"/>
        <custom>
            <javaType>java.lang.String</javaType>
        </custom>
    </type>
 
The values of child types may support the values of their direct parent type, or those of any ancestor type. Such a hierarchy enables actions that take only short, only long, or either variant of zip code as parameters.

Enumeration Types

Enumeration types allow for a fixed set of string values. Use of an enumeration type will cause the Adept UI and Editor to display values of that types as drop-down lists. For example:

    <type id="SizeEnum">
        <description>Enumeration of size types</description>
        <enum>
            <value>Small</value>
            <value>Medium</value>
            <value>Large</value>
        </enum>
    </type>        

Collection Types

Data types can be aggregated into sets, lists, or bags, which have an arbitrary number of elements, all of the same type. For example:

    <type id="set(string)">
        <description>A set of strings</description>
        <set>
	    <ref typeRef="string"/>
        </set>
    </type>
This type describes a set with elements of type string. In general, the element type may be any other type.

Structure Types

Data types can also be aggregated into structures. A structure has a fixed number of named elements of varying types. An element type may be any other type. For example:

    <type id="file">
        <description>
            A file on a file system, consisting of a directory,
            a base file name, and an extension indicating file type
        </description>
        <struct>
            <ref name="directory" typeRef="directory"/>
            <ref name="file name" typeRef="basefilename"/>
            <ref name="extension" typeRef="extension"/>
        </struct>
    </type>
This type describes a reference to a file, consisting of a containing directory, the base name of the file, and its file extension.

A structure can be made opaque. For example:

    <type id="coordinate" opaque="true">
        <description>
            An x,y coordinate
        </description>
        <struct>
            <ref name="x" typeRef="real"/>
            <ref name="y" typeRef="real"/>
        </struct>
    </type>
Making a structure opaque affects parameter generalization as follows:
  1. An opaque structure is never constructed from its component elements; it is either supported in whole by another structure, or is made an input parameter.
  2. An opaque structure's elements may never support any other values; that is, no generalizations will include accessors to opaque structure elements.

Compound types and generalization control

As noted earlier, certain types permit specifying generalization preferences for their values. List types allow specifying a preference for generalizing element access of a singleton value to either only($List), first($List), or last($List), with only being the default:

    <type id="sortedList">
        <description/>
        <list>
            <generalizeSingleton method="first"/>
            <ref typeRef="string"/>
        </list>
    </type>
Structure and collection types may specify a generalization preference for unsupported and partially supported values:
    <type id="aStruct">
        <description/>
        <struct>
            <generalizeUnsupported preference="construct" maxInputs="2"/>
            <ref typeRef="string" name="field1"/>
            <ref typeRef="string" name="field2"/>
            <ref typeRef="string" name="field3"/>
        </struct>
    </type>
The default preference is parameterize, which means to generalize by making the entire value a parameter to the procedure rather than constructing it from its parts. If maxInputs is not specified, procedure parameters will be added for each element of the collection or structure, regardless of number.

Metadata for data types

Allowable keys and values for type metadata are shown in the following table.

Metadata for data types
Key Value Applicable to Description

pluralName

string

all types

The plural of the type (e.g. "emails").

aName

string

all types

The type preceded by its indefinite article (e.g. "an email").

thisName

string

all types

The indefinite form of the type (e.g. "this email").

editor_disallowAskUser

boolean

all types

Disables the "Ask the user for this value" option on terms of this type (see the section "Rewiring Values in Steps" in the Editor Guide).

editor_disallowFixedValue

boolean

all types

Disables the "Use a fixed value" option on terms of this type (see the section "Rewiring Values in Steps" in the Editor Guide).

editor_disallowExistingValue

boolean

all types

Disables the ability to reference other variables on terms of this type (see the section "Rewiring Values in Steps" in the Editor Guide).

Actions

An action consists of

  1. a name that uniquely identifies the action,
  2. input and output parameters,
  3. [optional] a description of the action,
  4. [optional] metadata,
  5. [optional] an action category.
For example:
    <action id="OpenImageFile">
        <description>
           Opens an image file in the Image Viewer.
        </description>
        <metadata key="name" value="Open image in ,#file"/>
        <metadata key="fancyName" value="Open ,#file"/>
        <metadata key="icon" value="./fapps/image.jpg"/>
        <inputParam id="file">
            <description>
                The image file to open
            </description>
            <typeRef typeId="file"/>
        </inputParam>
        <outputParam id="image">
            <description>
               Identifies the in-memory image currently being manipulated
            </description>
            <typeRef typeId="imageID"/>
        </outputParam>
    </action>
The identifier is used internally to identify the action. If no naming metadata is supplied, the identifier is used in user interfaces to depict the action as well.

The input parameters describe the data needed by the action to execute. The output parameters describe the data that is computed or produced by the action. Put another way, they describe the dataflow of the action. An action with no outputs is permissible. This indicates that no relevant dataflow for the action exists, i.e., there are no significant data side effects. An action may also have no inputs. For example, an action to report the current date might have as output parameters the year, month, and day.

Parameters and dataflow completion

Input parameters may specify an optional class. The default value is generalizable, which may be explicitly specified if desired.

Input parameters may be specified as ungeneralizable. This means that the value used for that parameter in a learned procedure will be identical to the corresponding value in the demonstration from which it is learned. For example, with the class="constant" attribute on the "Format" parameter, the following action definition specifies that the format of the file produced by this action in a learned procedure will always be the same as the format observed in the demonstration.

  <action id="Convert">
    <description>
      Converts a file to the given format, producing a file with that extension.
    </description>
    <inputParam id="Infile">
      <description> The file being converted </description>
      <typeRef typeId="string"/>
    </inputParam>
    <inputParam id="Format">
      <description> The format the file is converted to. </description>
      <class class="constant"/>
      <typeRef typeId="string" />
    </inputParam>
    <outputParam id="Outfile">
      <description> The converted file. </description>
      <typeRef typeId="string"/>
    </outputParam>
  </action>

Metadata for actions

Allowable keys and values for action metadata are shown in the following table. The action model schema expects that all metadata tags be located after the description tag.

Metadata for actions
Key Value Applicable to Notes

name

string

all actions

A short descriptive name for the action, suitable for display for the user (e.g. "Open document")

fancyName

string

all actions

A descriptive name for the action that may contain references to action parameters of the form ,#paramName, (e.g., "Copy ,#from, to ,#to"). The paramName must either be followed by a comma, or must appear at the end of the string. When displaying the demonstration, each parameter reference is replaced with the corresponding demonstrated value. When displaying the learned procedure, each parameter reference is replaced with the corresponding variable or expression assigned by LAPDOG.

icon

resource

all actions

A path to a resource that is an icon file that depicts this action (e.g., "./resources/image.jpg"). Note that this file must be available on the classpath.

Allowable keys and values for action parameter metadata are shown in the following table.

Metadata for parameters of actions
Key Value Applicable to Notes

userDescription

string

all actions

A user-oriented description of the parameter.

Action categories and dataflow completion

Actions may specify an optional category. The default value is effector, which may be explicitly specified if desired. The category may also be completer, supporter, or context. This allows specification of dataflow completion actions as discussed in the Dataflow Completion section. For example, this action is from the action model of Image Loader:
    <action id="DefaultFolder" category="completer">
        <description>
            Add this action automatically when the designated default directory is demonstrated.
	    This means the learned procedure has one less parameter, but one more action.
        </description>
        <metadata key="name" value="Use default Imageloader folder"/>
        <metadata key="fancyName" value="Use default Imageloader folder"/>
        <metadata key="icon" value="./fapps/image.jpg"/>
        <outputParam id="defaultFolder">
            <description>
                The directory designated as the default folder for Imageloader
            </description>
            <typeRef typeId="directory"/>
        </outputParam>
    </action>
With this action, the system can infer support for the use of this default directory in the demonstration and add an instance of this action to the learned procedure to support that value. The net effect is that the default directory does not become a procedure parameter.

Action Modeling for Multiple Applications

When using Adept to create learned procedures that span multiple applications, each application must have its own action model, organized in separate files. The action models are independent of each other, except in those cases in which it is desirable to share dataflow across applications.

Sharing dataflow between two applications A and B means that the output of an action of A is used as an input to an action of B. As discussed in the Support Relationships section, this means that the types of this output and input must be compatible. This is a problem because by default the types of different action models are disjoint. However, Adept permits types in different action models to be specified as equivalent to each other, which makes them compatible for determining support relationships. For example, if application A's action model defines a type url:

    <type id="url">
        <description>A url such as http://www.yahoo.com</description>
	<!-- Allow URL params from application B to support and be supported by our URLs -->
	<equivalentTo>B.url</equivalentTo>
        <custom>
            <javaType>java.lang.String</javaType>
        </custom>
    </type>
the equivalentTo specification permits the url type of application B to support and be supported by the url type of application A. The javaType of equivalent types must be identical.

Design Guidelines

The action model is central to integrating an application with Adept. Guidelines for creating an effective action model are provided in this section.

Granularity

Representing actions at an appropriate level is key to creating understandable and manageable learned procedures. Actions should be modeled at the level at which humans would typically describe their own interactions with the client. For example, in an email application, capturing the actions Open Compose Window, Add Email Attachment, Send Email, etc. is preferable to Click on Compose Button, Pop Up Compose Window, Click on Attach Button, etc., and even more so to Drag Mouse from (X1,Y1) to (X2,Y2), Left-Click on Mouse at (X2,Y2), etc.

The system need not necessarily be directly instrumented/automated at a human-friendly level if a well-defined mapping exists between the system's native instrumentation and the higher-level conceptualization of actions used in the action model. For example, an email application might use an internal operation SendMail that handles sending a new message, replying, and forwarding all in one API, by providing the appropriate parameters for to and cc lists, message bodies, etc. An action model designer might prefer separate actions for Send, Reply, and Forward. This is accomplished by capturing SendMail calls for instrumentation, determining which of the three actions apply given the parameters of the SendMail, and appropriately mapping the parameters to the action's parameters. The reverse would be done for automation.

Complex Objects as Parameters

An arbitrary Java object is a bad candidate for a parameter for an action. The Adept UI does not support display and input of arbitrary Java objects. Also, the Java class of any object passed as a parameter in an action must fulfill the following: For these reasons, an alternative to complex objects as parameters might be desirable. One alternative is to define a unique identifier string for each object of a given class, and then map the object to its identifier when producing instrumentation and vice versa when processing automation.

Another alternative is to model the complex object as a structure. This is particularly appropriate if the object has visible data fields, as opposed to being a "black box" for which only the internals of the client application can access its components.

Dataflow Model

The effects of executing an action are characterized by their inputs and outputs. That is, in a strict dataflow model, the only meaningful state changes that occur within an application as a result of executing an action are those changes reported in its output parameters.

The dataflow model can pose problems in those cases in which an action changes a parameter or an attribute of a parameter, rather than producing a distinct output. If expressing the operations of a client application in terms of dataflow is infeasible, then the generalizations performed by task learning will be of limited value.

Iteration

Adept has the powerful capability to learn procedures with loops. Specifically, Adept can learn loops over lists, sets, and bags of values by recognizing patterns of actions that use their elements in a consistent way. This is discussed in the previous section under Loop Learning.

In order to learn loops at all, at minimum a list, set, or bag must appear in a demonstration as the output of an action. If iteration is important for an application, then its action model must provide the aggregates to be iterated over.

To learn a loop over a list of values, those values must appear in the demonstration in the same order in which they appear in the list. This may affect decisions on whether to model certain parameters as lists versus sets or bags.

Instrumentation and Automation Duality

With the exception of completer and context actions, every action in the action model must enable both reporting when the user performs that action (instrumentation) and executing that same action (automation) from a learned procedure. (Completer and context actions need only provide automation.) The effect of automation for an action should be exactly the same as executing the corresponding command natively in the client application.

Enabling the Client Application

This section explains how to create a module that allows a client application to exchange instrumented events and automated actions with the Adept task learning system. Conceptually, we will be using the Bridge APIs to make a connection between the client application and Adept. The JMS messaging spine used by Adept provides the communication between the two.

The APIs referenced in this section are implemented in the various files in the lib directory provided in the Adept installation. All files in this directory must be on the classpath of the client application to compile and execute it.

Numerous excerpts from sample Adept-enabled applications are provided. The complete source code of the Image Loader and Novo sample applications is provided as well.

Logging and Debugging

Adept uses log4j to create debugging logs. The client application may wish to enable log4j logging as well. This may be done by invoking the following Adept API static method in the package com.sri.tasklearning.util in its startup code, before any other Adept API methods are executed:

void LogUtil.configureLogging(String moduleName, Class baseClass)

Configure the log4j system with a config file. A series of config file names are searched for in the current directory, followed by looking for resources on the classpath relative to baseClass. Config files can be named moduleName.ext, where ext is one of xml, properties, or ini. As a fallback, log4j.ext will be searched for. If all else fails, this will try to use org.apache.log4j.BasicConfigurator.

Parameters:
moduleName - base name of the config file to be searched for
baseClass - class to search for resources relative to

For example, the Image Loader configures logging as follows:

  // use log4j.xml as logging config file.
  LogUtil.configureLogging("imageloader_log", Imageloader.class); 
  LogUtil.configureLogging("log4j", AdeptShell.class); 

This initialization allows a configuration file imageloader_log.xml to be included as a resource in its JAR file, or placed in its current working directory.

Initialization

First, we first create a class that is responsible for connecting to Adept and loading the action model. A typical class definition might look something like this:

import com.sri.pal.ActionModel;
import com.sri.pal.Bridge;
import com.sri.pal.PALException;
/**
 * AdeptWrapper acts as an interface to Adept, allowing access to functionality
 * such as instrumentation, type instantiation, and setting up the initial state
 * of Adept by loading an action model.
 */
public class AdeptWrapper {
    // Objects related to loading the actionmodel and executing
    // procedures
    public static final String NAMESPACE = "imageloader";
    public static final String APP_NAME = "imageloader";
    public static final String XML_PATH = "imageloader_model.xml";
    private static ActionModel actionModel;
    private static AdeptExecutor executor;
    public static Bridge bridge;
    private static boolean bridgeStarted = false;

    public AdeptWrapper() {
        loadActionModel();
    }
...
}
The NAMESPACE constant defines the name space in which the type and action names for this application's action model reside. The APP_NAME constant identifies the application. The XML_PATH is used to read the action model, which is done by the lone constructor.

The loadActionModel() method is used to load the action model from an XML file. A simple implementation might look like this:

    private void loadActionModel() {
        try {
            Bridge bridge = Bridge.newInstance(APP_NAME);
            actionModel = bridge.getActionModel();
            URL url = SomeClass.class.getResource(XML_PATH);
            executor = new AdeptExecutor();
            do {
                bridge = Bridge.newInstance(APP_NAME);
                actionModel = bridge.getActionModel();
                Set types = actionModel.load(url, NAMESPACE);
                for (ActionModelDef type : types) {
                    if (type instanceof ActionDef
                            && !(type instanceof ProcedureDef)) {
                        actionModel.registerExecutor(
                                (SimpleTypeName)type.getName(), executor);
                     }
                }
           } while (!Bridge.isTaskLearningRunning());
         } catch (PALException e) {
            log.error("Failed to load action model: ", e);
         }
    }

In this example, the class loader getResource() method is used to access the action model file. This means that the action model should be either in the application's classpath or contained in the module's jar file. Another option is to load the model from a common directory location. Next we call the actionModel.load() method passing in our XML file URL, the ActionExecutor (more on this later), and the application's namespace. The net effect is to connect to Adept via its Bridge interface and to initialize the action model from a file.

The application is responsible for indicating to Adept that it is responsible for executing each of its primitive actions. This is done by the loop after actionModel.load() by calling actionModel.registerExecutor() for each action in the loaded action model.

One shortcoming of this example is that it fails if Adept is not executing. The application may require a standalone mode, in which it operates normally but without any Adept interactions. The following example illustrates how to do so.

    /**
     * Attempts to establish the Image loader as a client to Adept. If it fails
     * Image loader can still continue running.
     */
    public void loadActionModel() {
        if (bridge == null) {
            Thread t = new Thread() {
                public void run() {
                    try {
                        Bridge.setConnectAttemptDuration(15);
                        Bridge.setConnectRetryInterval(5);

                        URL url = AdeptWrapper.class.getResource(XML_PATH);
                        executor = new AdeptExecutor();
                        do {
                            bridge = Bridge.newInstance(APP_NAME);
                            actionModel = bridge.getActionModel();
                            Set types = actionModel.load(url, NAMESPACE);
                            for (ActionModelDef type : types) {
                                if (type instanceof ActionDef
                                        && !(type instanceof ProcedureDef)) {
                                    actionModel.registerExecutor(
                                            (SimpleTypeName)type.getName(), executor);
                                }
                            }
                            Thread.sleep(5000);
                        } while (!Bridge.isTaskLearningRunning());
                        bridgeStarted = true;
                    } catch (PALException e) {
                        log.error("Failed to load action model: ", e);
                    } catch (InterruptedException e) {
                        log.error("Thread interrupted: ", e);
                    }
                }
            };
            t.start();
        }
    }
The technique used is to time out if repeated connections to the Bridge fail. This example attempts a connection every five seconds, giving up after 15 tries.

loadActionModel() should be executed when the client application starts executing. Adept must be running at this time in order to connect to it.

Instrumentation

The purpose of instrumentation is to report actions to Adept that capture all operations of the client application that are covered by the action model. Several coding styles are appropriate for doing so.

One style is to implement an event loop that responds to events generated by the application that correspond to user operations, dispatching on individual events. This is appropriate when a convenient event loop already exists in the client application, or if its interface to Adept resides in another process or thread. For example, an event capturing the opening of a URL by the Mozilla Firefox browser is represented by the following action:

    <action id="OpenUrlEvent">
        <description>Firefox open url event</description>
        <metadata key="name" value="Open URL"/>
		<metadata key="fancyName" value="Open: ,#url"/>
        <metadata key="icon" value="./fapps/icon_firefox.png"/>
        <inputParam id="url">
            <description>URL</description>
            <typeRef typeId="url"/>
        </inputParam>
    </action>
Instrumentation code must extract the relevant data from the event and invoke the correct action with the Adept system. Here is an example of the event handling:
        if (event instanceof IOpenUrlEvent) {
            IOpenUrlEvent openUrlEvent = (IOpenUrlEvent) event;
            String url = openUrlEvent.getUrl();
            TypeName openUrlName = new TypeName("OpenUrlEvent", NAMESPACE);
            ActionDef openUrlActionDef = (ActionDef) actionModel.getType(openUrlName);
            try {
                // Create an action invocation with no parent invocation,
                //   and a single URL parameter.
                ActionInvocation openUrlAction = openUrlActionDef.invoke(null, url);
                // Output parameters would be set here, if there were any.
                // Indicate action has completed.
                openUrlAction.setStatus(ActionInvocation.Status.ENDED);
            } catch (PALException e) {
                log.error("Failed to invoke open url action", e);
            }
        }
The IOpenUrlEvent is passed from the application to Adept. It has one input parameter, namely the URL that was opened. It does not have any direct relationship with the action model; the instrumentation code must map it to the OpenUrlEvent action.

First, we dispatch on the event's class. Next, we cast our event as the correct type. Then, we extract the URL parameter from the event. Finally, we create an action definition and invoke it. The action definition is obtained from the action model by calling the getType() method. Note that the type name is a combination of the event name from our XML file and the namespace used above. After we have our action definition, we call the invoke() method, passing in the URL. Note that this method takes a variable number of arguments and that order is important. The order of the arguments should correspond to the order of the corresponding parameters in the action model. After invoking the action, we need to set the status to ENDED.

A second alternative is to intersperse code, similar to the above, for each action throughout the client application at the point at which its execution occurs.

A third alternative requires invoking a utility method such as the following at each point in the client application at each point of action execution.

    /**
     * Takes an action name along with its arguments and invokes it so that
     * its instrumentation is collected by Adept. Input variables should come 
     * before output variables. This function does not accept null arguments or fewer
     * arguments than what is defined for the given action in the actionmodel.
     */
    public void instrumentAction(String name, Object... argVals) {
        if(!bridgeStarted) {
            log.warn("Could not instrument {}: learning not running.", name);
            return;
        }
        ActionDef def = (ActionDef) getType(name);
        ActionInvocation action = null;

        // set input arguments
        Object[] inputs = new Object[def.numInputFields()];
        int index = 0;
        for (; index < def.numInputFields(); index++) {
            inputs[index] = argVals[index];
        }

        try {
            action = def.invoke(null, inputs);
        } catch (PALException e) {
            log.error("Action invocation failed: ", e);
            return;
        }

        // set output arguments
        for (; index < argVals.length; index++) {
            action.setValue(index, argVals[index]);
        }
        action.setStatus(ActionInvocation.Status.ENDED);
    }
The client application calls this method with the appropriate arguments at each point that instrumentation is to be captured. A call looks like:

    if (instrumentation) {
	...
	someObject.instrumentAction("SelectImageFiles", dirPath, formatSet, fileSet);
    }

All of these instrumentation techniques rely on the com.sri.pal.ActionDef class and its invoke() method:

   /**
     * Creates a new invocation of this action definition.
     *   parent - always null when creating instrumentation
     *   args - input arguments, in the order in which they
     *          appear in the action model
     */
    public ActionInvocation invoke(ActionInvocation parent,
                                   List args)
            throws PALException;
An ActionDef represents all the information pertaining to an action that is specified in the action model for it. The invoke() method creates an invocation, indicating to Adept that the client has executed that action with the given input parameters. If the action has outputs, those values are filled in one at a time with the setValue() method. The first output goes after the last input, and must be in the order in which they appear in the action model. Finally, the status of the action is updated, indicating to Adept that its execution is complete. This completes instrumentation for this action.

Adept Types

There are two families of types in Adept, namely data types and action types. Each data type and action in the action model has a corresponding Adept type. An Adept type is represented by the com.sri.pal.TypeDef class.

To use several API methods, the client application needs to map the type names specified as IDs in the action model to the object representing that type. The following helper method may be used to do so for both data types and action types:

    /**
     * Requests a type from Adept given a type name.
     */
    public TypeDef getType(String typeName) {
        return actionModel.getType(new TypeName(typeName, NAMESPACE));
    }

Creating Adept Parameter Values

To create instrumentation, mapping the Java objects used by the application to their Adept counterparts is necessary. These are then passed to ActionDef.invoke(). If the Adept type is an atomic type, then no conversion is needed. For example, if the Adept type of an action parameter is
    <type id="integer">
        <description>An integer value</description>
        <custom>
            <javaType>java.lang.Integer</javaType>
        </custom>
    </type>
then any java.lang.Integer object is valid.

To use the various structured types of Adept—sets, lists, bags, and structures—care must be taken to create values with the proper structure. Sets, lists, and bags can use any of the Set or List implementations in the java.util.Collection framework. For example, in this method that creates a set of strings, a HashSet<String> is used to represent a set of strings:

   /**
     * Returns a Set of Strings, suitable for use as an action argument,
     * given the type name and a list of its elements.
     */
    public Set<String> createStringSet(String name, ArrayList<String> args) {
        if(!bridgeStarted) {
            log.error("Could not create set {}: learning not running.", name);
            return null;
        }
        Set<String> setValue = new HashSet<String>();

        for (String arg : args) {
            setValue.add(arg);
        }
        return setValue;
    }
To create structures, the com.sri.pal.StructDef class is used, which represents the information in the action model about a structure type. The structure value is represented by com.sri.pal.Struct. For example:
    import com.sri.pal.Struct;
    import com.sri.pal.StructDef;
    /**
     * Returns a Struct, suitable for use as an action argument,
     * given a type name and values for each structure field.
     * Arguments must be in the order the fields appear in
     * in the action model.
     */
    public Struct createStruct(String name, Object... args) {
        if(!bridgeStarted) {
            log.warn("Could create structure {}: learning not running.", name);
            return null;
        }
        StructDef structDef = (StructDef) getType(name);
        Struct structureValue = new Struct(structureDef);
        for (int i = 0; i < structureDef.size() && i < args.length; i++) {
            structureValue.setValue(i, args[i]);
        }
        return structureValue;
    }
Creating a set of structures is similar to creating a set of some primitive type:
    /**
     * Returns a Set of Structs, suitable for use as an action argument,
     * given a type name, and a list of Struct values.
     */
    public Set<Struct> createStructSet(String name, ArrayList<Struct> args) {
        if(!bridgeStarted) {
            log.warn("Could not create set {}: learning not running.", name);
            return null;
        }
        Set<Struct> setValue = new HashSet<Struct>();

        for (Struct arg : args) {
            setValue.add(arg);
        }
        return setValue;
    }

Automation

To implement automation, a class is created that implements the com.sri.pal.ActionExecutor interface. This interface requires a method that executes any incoming execution requests delivered by Adept. Adept delivers these requests in the form of an ActionInvocation, which is simply an action with its input parameters filled in. The execute() method dispatches on the action name, invoking the appropriate client-application code for each action and computing the values of any output parameters of the action. These output parameter values are then copied into the ActionInvocation. When execute() completes, Adept passes the completed ActionInvocation back to the requestor.

Here is the body of an example executor class. Note the idiosyncratic construction of the action names.

/**
 * AdeptExecutor implements the ActionExecutor interface in order to act as a
 * listener for when actions are executed from Adept. It handles five types of
 * actions from imageloader_model.xml: SaveImageFile, OpenImageFile,
 * ResizeImage, SelectImageFiles, and OpenNextImageFile. When it receives one of
 * these actions it extracts the input arguments, if any, sends the arguments to
 * ImageloaderGUI to execute, and then sets the output arguments generated, if
 * any.
 */
public class AdeptExecutor implements ActionExecutor {
    private static final Logger log = LoggerFactory
            .getLogger(AdeptExecutor.class);

    private ImageloaderGUI gui;
    public static final String NAMESPACE = AdeptWrapper.NAMESPACE;

    // Action names
    private static final String SAVE = "TypeName[" + NAMESPACE
            + ".SaveImageFile]";
    private static final String OPEN = "TypeName[" + NAMESPACE
            + ".OpenImageFile]";
    private static final String RESIZE = "TypeName[" + NAMESPACE
            + ".ResizeImage]";
    private static final String SELECT = "TypeName[" + NAMESPACE
            + ".SelectImageFiles]";
    private static final String NEXT = "TypeName[" + NAMESPACE
            + ".OpenNextImageFile]";

    public AdeptExecutor() {
        gui = ImageloaderGUI.getInstance();
    }
}

Here is its execute() method. Note the use of gui.endInstrumentation() and gui.startInstrumentation() methods to disable instrumentation during execution, which simply reset and set a flag in the client application:

    @Override
    public void execute(ActionInvocation arg0) throws PALException {
        if (arg0.getParentInvocation() == null) {
            log.error("Action execution failed: no parent invocation. ", arg0
                    .getDefinition().getName());
        }

        // make sure action demonstrations are not duplicated
        gui.endInstrumentation();

        int size = arg0.getDefinition().size();
        ArrayList<Object> items = new ArrayList<Object>();
        for (int i = 0; i < size; i++) {
            items.add(arg0.getValue(i));
        }

        // SaveImageFile
        if (SAVE.equals(arg0.getDefinition().getName().toString())) {
            String imageID = (String) items.get(0);
            
            Struct image = (Struct) items.get(1);
            String str0 = (String) image.getValue(0);
            String str1 = (String) image.getValue(1);
            String str2 = (String) image.getValue(2);
            String path = str0 + "/" + str1 + "." + str2;

            gui.saveImageEvent(new Integer(imageID), new File(path));
        // OpenImageFile
        } else if (OPEN.equals(arg0.getDefinition().getName().toString())) {
            Struct image = (Struct) items.get(0);
            String str0 = (String) image.getValue(0);
            String str1 = (String) image.getValue(1);
            String str2 = (String) image.getValue(2);
            String path = str0 + "/" + str1 + "." + str2;
            
            int ID = gui.openImageEvent(new File(path));
            arg0.setValue(1, Integer.toString(ID));
        // ResizeImage
        } else if (RESIZE.equals(arg0.getDefinition().getName().toString())) {
            String imageID = (String) items.get(0);
            Integer x = (Integer) items.get(1);
            Integer y = (Integer) items.get(2);

            int newID = gui.resizeImageEvent(new Integer(imageID), x, y);
            arg0.setValue(3, Integer.toString(newID));
        // SelectImageFiles
        } else if (SELECT.equals(arg0.getDefinition().getName().toString())) {
            String directory = (String) items.get(0);
            Set<String> formats = (Set<String>) items.get(1);
            ArrayList<String> list = new ArrayList<String>();
            list.addAll(formats);

            ArrayList<Struct> files = gui.selectImagesEvent(directory, list);
            arg0.setValue(2, files);
        // OpenNextImageFile
        } else if (NEXT.equals(arg0.getDefinition().getName().toString())) {
            Set<Struct> structureSet = (Set<Struct>) items.get(0);
            ArrayList<File> files = new ArrayList<File>();
            for (Struct t : structureSet) {
                String str0 = (String) t.getValue(0);
                String str1 = (String) t.getValue(1);
                String str2 = (String) t.getValue(2);

                String path = str0 + "/" + str1 + "." + str2;
                files.add(new File(path));
            }
            
            Struct image = (Struct) items.get(1);
            String str0 = (String) image.getValue(0);
            String str1 = (String) image.getValue(1);
            String str2 = (String) image.getValue(2);
            String path = str0 + "/" + str1 + "." + str2;
            
            int imageID = gui.nextImageEvent(files, new File(path));
            arg0.setValue(1, Integer.toString(imageID));
        } else {
            log.debug("Action {} ignored.", arg0.getDefinition().getName());
        }

        gui.startInstrumentation();
}

Instrumentation and Automation of Learned Procedures

Learned procedures are modeled as actions by Adept and are processed in exactly the same way as the primitive actions within an action model. In particular, these learned actions may be demonstrated during learning and consequently may be included in a learned procedure.

The application delivers instrumentation to Adept for all modeled actions that are executed, including actions that are executed from within a learned procedure. The application need not perform special processing to handle this case; Adept recognizes when an action originates from within a learned procedure and excludes that action from the demonstration, but includes instrumentation for all top-level learned procedures.

Persistence of Learned Procedures

Adept stores learned procedures in the client file system under the conventional, system-dependent directory for application data. For Windows systems, this is the user's "Application Data" folder (that is, wherever %APPDATA% refers to):

Adept handles all details of saving procedures from its user interface. No client programming is needed to enable it.