A breve bandirò un assegno di ricerca annuale per lavorare al progetto Axiometrics: Foundations of Evaluation Metrics in IR. Il progetto è finanziato da un Google Faculty Research Award (http://research.google.com/university/relations/research_awards.html).
Abstract del progetto Axiometrics
Effectiveness evaluation is of paramount importance in the field of Information Retrieval (IR). IR is probably the most evaluation-oriented field in computer science, as witnessed by an evaluation methodology developed in the 60s during the Cranfield experiments and by several evaluation initiatives running today (TREC, CLEF, NTCIR, INEX, FIRE). One crucial aspect of evaluation are evaluation metrics. About 100 IR effectiveness metrics exist, and counting. This project aims at understanding the relationships among them, in terms of both axiomatic properties and statistical relations, for both metric science (understanding of metrics) and engineering (their development). More in detail, we aim at proposing:
- Axioms: rules that any metric must satisfy. For example, when swapping a relevant document and a non-relevant one in the ranking, by decreasing the rank of the relevant one and increasing the rank of the non relevant one, the metric value should decrease. Axioms might be verified and compared with user intuition using crowdsourcing.
- Desiderata: desirable properties that a metric should have, based on common sense. For example, an effectiveness value according to one metric should be affected more by a swap in earlier rank positions than a swap in later ranks.
- Empirical properties: those that emerge from data, i.e., from actual test collections and system comparisons. These include robustness, statistical correlation between metrics, etc.
Axiometrics è una delle linee di ricerca più importanti per l’IR proposte durante il meeting SWIRL 2012 (http://www.cs.rmit.edu.au/swirl12/). L’attività è svolta in collaborazione con:
- Stefano Mizzaro (Principal Investigator), Dept. of Maths and Computer Science, University of Udine, Italy, mizzaro_pippo@uniud.it.
- Julio Gonzalo (co-Principal Investigator) ed Enrique Amigó, E.T.S.I. Informática de la UNED, Madrid, Spain, julio_pippo@lsi.uned.es, enrique_pippo@lsi.uned.es.
- Evangelos Kanoulas (Google sponsor), Google Zurigo, ekanoulas_pippo@gmail.com.
[ovviamente gli indirizzi email contengono alcuni caratteri in più da rimuovere, a meno che tu non sia uno spammer :)]
Se pensi che ti possa interessare, contattami.
Stefano Mizzaro
www.dimi.uniud.it/~mizzaro
