As part of our work developing a problem-solving method for plan evaluation , we are developing an ontology of evaluations and critiques of plans. The diagram below shows the most important entities and relations in this ontology and, on the right, runs through a simple example expressing a critique.
We can make evaluations of plans for a number of different reasons. In one case we may draw together a set of related features of a plan in order to gain a picture of the plan's essential structure, the things that make it qualitatively different from another possible plan. For example we may look at a plan's use of resources and its assumption of risk. These features may either be deduced or inherent in the plan's definition. In our ontology, the evaluation structure groups together related analyses, each of which mentions some object property, such as fuel usage, and one or more estimates of a value. The analyses can have more than one estimate because the evaluation system may have more than one algorithm at its disposal for providing an estimate. For example, a system may begin with a gross estimate of fuel usage based on heuristics, and then add a finer estimate based on a detailed simulation of the plan.
We often want to associate an analysis of a plan with some normal or expected value for that analysis, in order to see if the plan deviates from a norm. In our ontology a specialization of an analysis, called an evaluation, provides this capability. It inherits an object property and a set of estimates as a kind of analysis, but also has a normal value and a comparison structure. For instance, we might expect some plan to use 100 gallons of fuel and find that it actually uses 150 gallons, giving a comparison ratio of 3:2. Sometimes in evaluating a plan we want to be alerted if some evaluation has exceeded certain limits from its normal value, and this is expressed in our ontology by critique, which is a kind of evaluation. This entity has a norm, that typically indicates a range around the normal value that is acceptable, and a degree of violation, that records by how much, if at all, the critique is violated. For example, we may state that fuel usage may deviate from the normal amount by 15% without our concern, but higher levels of deviation represent a violation.
Click the diagram to see a larger version.
HPKB project members can access the ontology in Loom and KIF (password protected).
If you would like a copy of the evaluation ontology or would like to discuss this work, please contact Jim Blythe, by email to [email protected].