Postdoc position

In the context of a Google Faculty Research Award (http://research.google.com/university/relations/research_awards.html), we are seeking a post doctoral researcher at Udine University (Udine, Italy). The title of the project is: 

    Axiometrics: Foundations of Evaluation Metrics in IR. 

Axiometrics is one of the most important research directions proposed during the SWIRL 2012 meeting (http://www.cs.rmit.edu.au/swirl12/). A slightly more detailed description is at the end of this announcement. This is a joint project, involving:

• Stefano Mizzaro (Principal Investigator), Dept. of Maths and Computer Science, University of Udine, Italy, mizzaro_foo@uniud.it.

• Julio Gonzalo (co-Principal Investigator) ed Enrique Amigó, E.T.S.I. Informática de la UNED,  Madrid, Spain, julio_foo@lsi.uned.es, enrique_foo@lsi.uned.es.

• Evangelos Kanoulas (Google sponsor), Google Zurigo, ekanoulas_foo@gmail.com.

[you'll need to remove some characters from the email address -- unless you're a spammer :)] Feel free to contact me for further information, informal enquires, etc. The position is for one year (renewable if further funds are available). Salary is around 1900 euros/month. The activity will start in May/June 2013. The formal announcement in Italian is at http://www.uniud.it/ateneo/normativa/albo_ufficiale/183-2013 (# 20). 

*** Application deadline: 16 April 2013. ***

Short project description

Effectiveness evaluation is of paramount importance in the field of Information Retrieval (IR). IR is probably the most evaluation-oriented field in computer science, as witnessed by an evaluation methodology developed in the 60s during the Cranfield experiments and by several evaluation initiatives running today (TREC, CLEF, NTCIR, INEX, FIRE). One crucial aspect of evaluation are evaluation metrics. About 100 IR effectiveness metrics exist, and counting. This project aims at understanding the relationships among them, in terms of both axiomatic properties and statistical relations, for both metric science (understanding of metrics) and engineering (their development). More in detail, we aim at proposing:

• Axioms: rules that any metric must satisfy. For example, when swapping a relevant document and a non-relevant one in the ranking, by decreasing the rank of the relevant one and increasing the rank of the non relevant one, the metric value should decrease. Axioms might be verified and compared with user intuition using crowdsourcing.

• Desiderata: desirable properties that a metric should have, based on common sense. For example, an effectiveness value according to one metric should be affected more by a swap in earlier rank positions than a swap in later ranks.

• Empirical properties: those that emerge from data, i.e., from actual test collections and system comparisons. These include robustness, statistical correlation between metrics, etc.

Stefano Mizzaro
www.dimi.uniud.it/mizzaro

Computer science education

Un post interessante...

S.