Continuous Terrain Modeling from Image Sequences
with Applications to Change Detection

Subcontractors

Title of Effort

Principle Investigator

Technical Staff

Technical Area

Approach

Abstract

The automatic monitoring of a large geographic area for meaningful changes on a continuing basis is a critical ability for any modern military command. SRI will develop methods to incrementally model and detect changes over a large geographic area from range data (such as interferometric synthetic aperture radar (IFSAR) elevation data) and electro-optical (EO) and infrared (IR) video data, such as that obtained from the Predator or Conventional High Altitude Endurance (Conv HAE) UAVs.

 In this project, meaningful changes refer to changes in either the shape of the terrain (due perhapes to bomb damage, movement of large machinery, deforestation, and so on) or in the surface material properties of the terrain (due perhaps to change in ground cover, pouring asphalt over a dirt road, building an air strip, and so on). Attributions of the terrain, such as delineations of roads and identifications of buildings, are beyond the scope of this project.

Visionary System Description

The year is 2001. The analyst receives an Air Tasking Order (developed 12 hours before the mission commences) to monitor the effects of a precision strike attack on a chemical facility. The Combat Assessment team following the targeting cycle ``First Look, First Shoot, First Kill'' philosophy wants to know the status of the targets before, during and after the attack (so they can retarget if necessary), as well as all collatoral damage (worker housing is within 300 meters of the facility). The facility will be under continuous surveillance from a Conventional High Altitude Endurance (Conv HAE) UAV. The analyst will have 10 minutes to report his/her finding so the additional backup aircraft can attack the target again if necessary. The attack aircraft's missiles are fitted with a camera that will provide a view of the attack angle and moment of impact. Six buildings will be attacked with two attack aircraft firing six Precision Guided Missiles each. Supporting the analyst are one chemical engineer and one structural engineer, neither is an experienced image analyst, and they will need high resolution visualization tools along with mensuration tools to determine the extent of damage to the facility and the effectiveness of the mission. The analyst working with the engineers determines what imagery and other data is required perform the task.

 Based on the task, the analyst requests imagery, maps, terrain analysis data, DTED, and collatoral engineering data of the facility through the United States Imagery System (USIS) Global Geospatial Information and Service (GGIS). Available are DTED Level 2 (30M postspacing, elevation accuracy in this terrain approximately 5 meters). Stereo EO imagery exists of approximately 1 meter resolution. IFSAR with a resolution of 0.3 meters covers the entire facility and surrounding area. 1:50,000 Vector Maps of the area are available. No site models are available. Engineering drawings of the facility are not available. Utility distribution drawings of the facility and worker housing area are available which show power and underground fuel lines inside and surrounding the facility.

 The analyst then goes to his/her analyst tool kit, and selects the tools needed to monitor the site before, during and after the attack. These tools include imagery registration tools, image perspective tranformation (IPT) tools, image processing tools, and (most importantly) our terrain modeling and change detection tool.

 First, maps, IFSAR elevation data, EO/IR imagery, SAR and DTED are combined to create a complete 3-D model of the shape and surface properties of the facility. The surface properties of the terrain are estimated from the imagery based on the known position of the sun, the shape of the terrain (taking shadows and occlusions into account), camera and radar parameters, as well as cloud cover and other relevant information. Known ``deficiencies'' of the sensors (such as occluded areas in EO/IR imagery or ``front-porch'' artifacts in IFSAR data) are used to rigorously derive error tolerances and covariances for every element of the model. This model becomes the baseline or reference model to compare changes against.

 Using the baseline terrain model as the reference scene, incoming UAV will be compared against the terrain model to detect changes in the terrain. It is the integrated 3-D nature of this representation and processing methodology that allows the system to rapidly detect changes in both the shape and surface material properties of the terrain. Changes in the terrain's shape are detected by comparing 3-D shape and material properties derived from incoming data against the model. For incoming UAV EO/IR imagery, the mesh-based terrain modeling algorithm is used to register and derive an updated 3-D model from the imagery. The derived model is then continuously compared against the current updated model, using the error tolerances mentioned above to detect areas of significant change during and after the attack.

 Prior to the attack any available new imagery or elevation data will be used to continuously refine the terrain model wherever the new imagery is consistent with the model (i.e., when the elevation data and surface properties derived from the new imagery are within the automatically derived error tolerance of the model). This will allow the model to become increasingly accurate and reliable over time. As the model becomes more accurate, it supports more sensitive change detection important to combat assessment where small, but important detail, is required.

 The gun camera video imagery is registered against the terrain model and is used to derive precise changes in the terrain for analysis. Areas of significant difference (after the attack) are highlighted on the both the initial and updated models that the analyst can view from any vantage point in 3-D. Various critical measurements of the terrain differences before and after the attack (e.g., volume of rubble, size of crater) are available to the analyst (who needs no skills in image understanding algorithms) by interactively selecting areas in the two models and having the system automatically compare them and perform the needed calculations.

Impact

The deployment of the Predator UAV will generate large quantities of SAR/IFSAR and EO/IR data of great value to Battlefield Awareness if it can be interpreted quickly and affordably. We will develop and demonstrate a system that will automatically generate and refine a 3-D model of the terrain's shape and surface properties from IFSAR, EO, and IR data, and detect changes in both elevation and surface properties. Such changes can then be noted on the model and reported to ground personnel (or other automated systems) for review and action. We thus expect to be able to dramatically reduce the amount of analyst time necessary to take advantage of UAV data.

Innovative Ideas

We propose a revolutionary method for detecting change that uses a new object-centered representation called a ``deformable mesh'' in which radar and EO/IR imagery taken at different times of day and from different points of view are combined into a unified 3-D model of the shape and surface properties of the terrain. The deformable mesh is not only a unified representation of the terrain but also a computational framework in which the processing of incoming data is performed. This unified framework is necessary to allow for the reliable detection of changes in either shape or material property, and is significantly more accurate and robust than traditional processing where, for example, independent depth maps are recovered from stereo pairs and the maps are then ``averaged'' together in some fashion. The method involves the following components.

 

 

Model Creation First, maps, IFSAR elevation data, EO/IR imagery, and other sources of information (e.g., terrain type, building models) will be combined to create a complete 3-D model of the shape and surface properties of the geographical area covered by the data. The surface properties of the terrain will be estimated from the imagery based on the known position of the sun, the shape of the terrain (taking shadows and occlusions into account), camera and radar parameters, cloud cover, and other relevant information. Known ``deficiencies'' of the sensors (such as occluded areas in EO/IR imagery or ``front-porch'' artifacts in IFSAR data) will be used to rigorously derive error tolerances and covariances for every element of the model.

 

 

Change Detection Second, new imagery and new IFSAR elevation data will be compared against the terrain model to detect changes in the terrain. It is the integrated 3-D nature of our representation and processing methodology that allows us to detect changes in both the shape and surface material properties of the terrain, as follows.

 Changes in the terrain's shape will be detected by comparing 3-D shape and material properties derived from incoming data against the model. This can be done directly for incoming IFSAR range data. For incoming EO/IR imagery, our mesh-based terrain modeling algorithm will be used to register and derive a new 3-D model from the imagery. This derived model will then be compared against the current model, using the error tolerances mentioned above to detect areas of significant change.
 
 

Model Refinement Third, new imagery and elevation data will be used to continuously refine the terrain model wherever the new imagery is consistent with the model (i.e., when the elevation data and surface properties derived from the new imagery are within the automatically derived error tolerance of the model). This will allow the model to become increasingly accurate and reliable over time. As the model becomes more accurate, it supports more sensitive change detection.

 

 

Model Extension Finally, incremental extensions of the model to new areas will be made wherever IFSAR range data or overlapping images cover a portion of the terrain that has not yet been modeled.

Major Project Deliverables

SRI will design and develop a demonstration system integrated in the RCDE, and evaluate experiments conducted in the system to establish the performance on data provided by the IFD contractor for this area. Because SRI is a developer of the RCDE, all software will be integrated within the RCDE, and will therefore be exportable to other groups using the same environment.

Military/Battlefield Relevance

Demonstrations Scheduled

Recent Publications

Recent Briefings

Relevant images(s) with captions

Links to additional sites that are relevant