%0 Report %A Barker, K. and Chaudhri V. and Clark, P. and Israel, D. and Porter, B. and Romero, P. %T Halo Pilot Project %I SRI International %D 2003 %X The Halo Pilot Project is the first phase of a projected multi-phase effort by Vulcan Inc. whose ultimate goal is the creation of a digital Aristotle, an expert tutor in a wide variety of subjects. The Halo Pilot was a six-month effort intended to assess the state-of-the-art in question answering, with an emphasis on deep reasoning. The program was structured around the challenge of responding to variants of AP Chemistry questions that focused on a portion of the AP Chemistry syllabus. Over a four-month period, we developed a question-answering system for Halo based upon SHAKEN, a Knowledge Representation and Reasoning system that the SRI team has developed in a series of extremely productive collaborations on DARPA-funded projects. This involved encoding the corpus in the formalism of SHAKEN (KM), developing problem solving and reasoning methods to answer the questions and creating a new answer explanation facility for producing concise and understandable explanations of the use of those methods to answer the questions. The system we developed was sequestered while we spent an additional two weeks encoding 150 mostly novel, that is, previously unseen chemistry questions. These questions were then processed on the sequestered system. . Vulcans methodology emphasized the following elements of such a challenge: (i) coverage, the systems ability to answer novel questions from the specified syllabus; (ii) question encoding, the ability to robustly create high-fidelity translations of the questions into SHAKEN, and (iii) explanation generation, the ability to provide concise and coherent explanations for the answers produced. In addition, in order to help direct future research in this area, the Halo participants developed brittleness" taxonomy of failure modes of KRR systems and each systems failures were classified into cells of that taxonomy. Two features, in particular, of the Halo Pilot were especially challenging: competence in chemistry, as in many of the physical sciences, depends crucially on the ability to reason with a large number of laws, at somewhat different levels of generality, combined in the right way and in the right order. Domains treated in previous projects have not had this character. Second, the ability of a system to produce coherent and concise explanations of its reasoning was central to the Halo Pilot, as it would be to the trustworthiness and hence utility of any complex software system.