TY - CONF TI - Novelty and Diversity in Information Retrieval Evaluation AU - Clarke, Charles L.A. AU - Kolla, Maheedhar AU - Cormack, Gordon V. AU - Vechtomova, Olga AU - Ashkan, Azin AU - Büttcher, Stefan AU - MacKinnon, Ian T3 - SIGIR '08 AB - Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track. C1 - New York, NY, USA C3 - Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval DA - 2008/// PY - 2008 DO - 10.1145/1390334.1390446 DP - ACM Digital Library SP - 659 EP - 666 LA - en PB - ACM SN - 978-1-60558-164-4 UR - http://doi.acm.org/10.1145/1390334.1390446 Y2 - 2019/01/27/19:15:02 ER - TY - CONF TI - An Axiomatic Approach for Result Diversification AU - Gollapudi, Sreenivas AU - Sharma, Aneesh T3 - WWW '09 AB - Understanding user intent is key to designing an effective ranking system in a search engine. In the absence of any explicit knowledge of user intent, search engines want to diversify results to improve user satisfaction. In such a setting, the probability ranking principle-based approach of presenting the most relevant results on top can be sub-optimal, and hence the search engine would like to trade-off relevance for diversity in the results. In analogy to prior work on ranking and clustering systems, we use the axiomatic approach to characterize and design diversification systems. We develop a set of natural axioms that a diversification system is expected to satisfy, and show that no diversification function can satisfy all the axioms simultaneously. We illustrate the use of the axiomatic framework by providing three example diversification objectives that satisfy different subsets of the axioms. We also uncover a rich link to the facility dispersion problem that results in algorithms for a number of diversification objectives. Finally, we propose an evaluation methodology to characterize the objectives and the underlying axioms. We conduct a large scale evaluation of our objectives based on two data sets: a data set derived from the Wikipedia disambiguation pages and a product database. C1 - New York, NY, USA C3 - Proceedings of the 18th International Conference on World Wide Web DA - 2009/// PY - 2009 DO - 10.1145/1526709.1526761 DP - ACM Digital Library SP - 381 EP - 390 LA - en PB - ACM SN - 978-1-60558-487-4 UR - http://doi.acm.org/10.1145/1526709.1526761 Y2 - 2019/01/27/22:06:28 ER -