Print

He, Jiangyan

Computer Science

2005

MS

This study conducts experiments on the Combination-of-Expert-Opinion (CEO) model to verify its theory on the improvement of document ranking performance. It is the first empirical study of the CEO model that involving testing on two hypothesis made in the theoretical model: 1) combining different text search systems overall achieves more precise relevance ranking than the individual ones, and 2) combining two "dissimilar" systems (those with the less correlation in the retrieved sets) achieves more precise relevance ranking than combining two "similar" systems (those with the more correlation in the retrieved sets). We chose TREC-4 collections comprising full text documents from congress records (CR93) and 50 queries (Topic 251 - Topic 300). We used three retrieval systems. The first two systems are "similar" to each other, but the third one is "dissimilar." The experiments showed that when two different systems are combined, both hypotheses proved to be valid overall given the systems and data sets used in the experiments. We also investigated combining all three retrieval systems to see if it improves the retrieval results further. However, no improvement was observed over the case of combining two dissimilar systems. That is, when three different systems are combined, the first hypothesis proved to be invalid overall. Our conclusion from the study is that CEO enables users to consider multiple factors in relevance distribution and feedback, thus overcoming the problems in older models like the RMC model (Robertson, Maron, and Cooper's model); on the other hand, our experiment results also suggest that the CEO model and its implementation need further refinements.