Automatic Population of Geospatial Databases
Test Plan for Evaluation Day at Terrain Week '98
Introduction
Terrain Week '98 is sponsored by DARPA, STRICOM, and TEC and will be
held on 18-22 May 1998 in Austin, Texas. Terrain Week lets
researchers meet and discuss common issues in an informal environment.
The emphasis is on sharing technical results, with a minimum of
programmatics. Additional details and registration can be found at
TEC's Terrain Week Web Site.
The first three days will be devoted to the APGD community.
Evaluation Day (18 May) will focus on the evaluation of feature
extraction results. SRI has produced a document entitled ``On the
Evaluation of APGD Algorithms and Systems'' which outlines a
proposed common evaluation methodology.
APGD researchers involved with road and building extraction will be
asked to participate in evaluations over the Ft. Benning McKenna MOUT
site and Fort Hood. The first cut at a common evaluation process and
results will be discussed at Evaluation Day. The focus will be on
the evaluation process and not the results.
Test Imagery and Reference Models
- The test imagery to be employed in this exercise has already been
distributed to all APGD community members. In particular, it includes
the FT Benning/McKenna MOUT data set EO collection, DEM and
Orthophoto:
-
http://www.ai.sri.com/~apgd/vl/datasets/Benning/
- and the Ft Hood collection:
-
http://www.ai.sri.com/~apgd/vl/datasets/Hood/
Areal Extent of Test and Features to be Extracted
Roads
For road delineation, it is expected that results will be produced for
the entire area of the McKenna MOUT site that is covered by the
orthomosaic and DTED that was distributed with the dataset. The
bounding box in Zone 16 UTM coordinates (WGS-84/MSL) for this area
are:
| UTM Zone 16 | Easting(x) | Northing(y) |
| Lower-left: | 705474.21 | 3582913.10 |
| Upper-right: | 708261.75 | 3585300.52 |
- The Reference Model is available for inspection or downloading at:
- http://www.ai.sri.com/~apgd/vl/datasets/Benning/road-ref-1.apgd
- The Reference Model projected on to the ortho imagery:
- http://www.ai.sri.com/~apgd/vl/datasets/Benning/eval-road-references.html
This is in the format described in Appendix B of
``On the Evaluation of APGD Algorithms and Systems''
Buildings
For building extraction, it is expected that results will be produced
for any (or all) of the three sites, the Ft. Benning McKenna MOUT
``Village'' and two areas within the motor pool area at Ft. Hood:
-
Ft. Benning Village Lower Left: (706678.56 3583429.09) Upper Right: (706396.65 3583719.04) Zone 16
- Ft. Hood 711 showing the two study areas
- Ft. Hood Area 1 Lower Left: (617448.35 3446163.69) Upper Right: (617754.28 3446473.19) Zone 14
- Ft. Hood Area 2 Lower Left: (617977.32 3445450.29) Upper Right: (618441.43 3445657.66) Zone 14
- Reference models for building extraction are available
for inspection at:
- http://www.ai.sri.com/~apgd/vl/datasets/Benning/ft-benning-ref-buildings.apgd
- http://www.ai.sri.com/~apgd/vl/datasets/Hood/ft-hood-buildings.apgd
This is in the format described in Appendix B of
``On the Evaluation of APGD Algorithms and Systems''
The Derived Model Evaluation Procedure
- Participants in the Terrain Week '98 evaluation exercise are expected
to use their algorithm/system to extract roads and/or buildings found
in one or more of the three test sites listed above and encode their
derived building or road models in the Evaluation Data Format
specified in:
- Appendix B of
``On the Evaluation of APGD Algorithms and Systems''
Options for performing the derived model evaluation include:
- Automatic (remote) evaluation of derived road-network models (MOUT
site) by following the instructions on the APGD WWW Virtual Lab at:
- http://yuba.ai.sri.com/cgi-bin/lisp/terrain-week.lisp
- Automatic (remote) evaluation of derived building models (MOUT site
and Ft. Hood) by following the instructions on the APGD WWW Virtual
Lab at:
- [TBD]
- Self-evaluation of his results by the participant using software
provided by SRI/GDE (this requires that the participant has an RCDE
system). This option may not be practical for terrain week, but could
be available for subsequent evaluations.
- Self-evaluation of his results by the participant using software that
duplicates the specifications in the document described below
(Proposed APGD Evaluation Metrics and Report Format"), and the
reference models downloaded from the APGD WWW Virtual Lab.
Format for Reporting Results
Section 2 of
``On the Evaluation of APGD Algorithms and Systems''
specifies the information to be provided by an APGD researcher participating in the
Terrain Week '98 exercise, and is proposed for general use within the
APGD community as a format for presenting experimental results
acquired in a benchmark evaluation of the performance of (road and
building) feature extraction algorithms.
The rationale for the requested information, choice of metrics, and
evaluation procedure is provided in Appendix A of the
document. Appendix B provides the detailed data formats and examples
of their use.