Representing probabilistic knowledge (e.g., of statistical relations) is a key challenge for OWL ontologies. OWL, being based on first order logic, does not have any inherent (or obvious) mechanisms for representing probabilities. There are two major sorts of approach to incorporating probabilities in logical theories:
- Knowledge Based Model Construction (KBMC) formalisms (e.g., P-CLASSIC, MEBN)
- (So-called Nilsson style) Probabilistic Logics (PL) (e.g., P-SHROIQ)
These approaches have fundamentally different approaches: KBMC theories tend to determine exactly one probability distribution (a la Bayesian Networks). That is, a syntactically correct theory captures at most one distribution and always captures one (i.e., such a theory can never be unsatisfiable). PL based theories tend to have many validating distributions or none (thus, probabilistic satisfiability (PSAT) is a significant reasoning problem).
Comparing different formalism within a tradition, much less across traditions, is notoriously difficult. A reasonable step toward principled comparison is to develop a set of challenge problems which can serve as a series of case studies for each formalism. While there are several repositories for Bayesian networks (which can be quite useful), they do not provide an independent specification of the modelling problem being tackled by each network. Thus, meaningful comparisons of models which do not express the exact meaning of the given network requires considerable additional knowledge, which may not be readily available or even exist.
Manfred Jaeger maintains a list of challenge problems for probabilistic formalisms. The criteria for the problems is somewhat informal:
A challenge problem can be given by one specific model that is encoded in a particular representation language (the “reference model”). Usually, these are model examples that come with the system implementing the language. The challenge for other languages/systems is to find an equivalent representation. Alternatively, a challenge can be formulated as a modeling task in general terms. The challenge problems are selected to be rather simple examples for fundamental modeling tasks.
There are comparatively few models and the non-learning challenges are primarily given in terms of a reference model.
Our goal is to develop a set of challenge problems which are more independent of a given modelling attempt and for which there are more or less formal evaluation criteria, but that these criteria are application based. Thus, equivalence to a given modelling is not especially interesting if the given modelling is not fit for an applications purpose.