Invited talks

I'll be giving two invited talks in the next months:

• Readersourcing: Scholarly publishing, peer review, and barefoot cobbler's children @ FDIA 2009 @ ESSIR 2009, 01/09/2009.
  
  Abstract:
  
  I will start from an introduction to the field of scholarly publishing, the main knowledge dissemination mechanism adopted by science, and I will pay particular attention to one of its most important aspects, peer review. I will present scholarly publishing and peer review aims and motivations, and discuss some of their limits: Nobel Prize winners experiencing rejected papers, fraudulent behavior, sometimes long publishing time, etc. I will then briefly mention Science 2.0, namely the use of Web 2.0 tools to do science in a hopefully more effective way.
  
  I will then move to the main aspect of the talk. My thesis is composed of three parts.
  
  (i) Peer review is a scarce resource, i.e., there are not enough good referees today. I will try to support this statement by something more solid than the usual anecdotal experience of being reject because of bad review(er)s --- that I'm sure almost any researcher has experienced.
  
  (ii) An alternative mechanism to peer review is available right out there, it is already widely used in the Web 2.0, it is quite a hot topic, and it probably is much studied and discussed by researchers: crowdsourcing. According to Web 2.0 enthusiasts, crowdsourcing allows to outsource to a large crowd tasks that are usually performed by a small group of experts. I think that peer review might be replaced --- or complemented --- by what we can name Readersourcing: a large crowd of readers that judge the papers that they read. Since most scholarly papers have many more readers than reviewers, this would allow to harness a large evaluation workforce. Today, readers's opinions usually are discussed very informally, have an impact on bibliographic citations and bibliometric indexes, or stay inside their own mind. In my opinion, it is quite curious that such an important resource, which is free, already available, used and studied by the research community in the Web 2.0 field, is not used at all in nowadays scholarly publishing, where the very same researchers publish their results.
  
  (iii) Of course, to get a wisdom of the crowd, some readers have to be more equal than others: expert readers should be more influential than naive readers. There are probably several possible choices to this aim; I suggest to use a mechanism that I proposed some years ago, and that allows to evaluate papers, authors, and readers in an objective way. I will close the talk by showing some preliminary experimental results that support this readersourcing proposal.
  
  Disclaimer: This talk might harm your career; don't blame me for that.
  
  P.S. Yes, this is somehow related to a previous post...
  
  
• Two Tales on Relevance Crowdsourcing: Criteria and Assessment @ GIScience Colloquium, University Zurich-Irchel, 13/10/2009.
  
  Abstract (DRAFT):
  
  In Information Retrieval (IR) and Web search, relevance is a central notion. I will discuss how to outsource to the crowd two relevance-related tasks. The first task concerns the elicitation of relevance criteria. After some results obtained in the 90es, relevance criteria (i.e., the features of the retrieved items that determine their relevance) seem well known and stable. We conjectured that for e-Commerce / product search the criteria might be different, and we used Amazon Mechanical Turk, a crowdsourcing platform, to find a confirmation of our hypothesis.
  
  The second task concerns effectiveness evaluation. A common evaluation methodology for search engines and IR systems is to rely on a benchmark (a.k.a. test collection); benchmarks need relevance assessment, i.e., to assess the relevance of documents to information needs; usually this task is done by experts, either paid for their work or participating in the evaluation exercise themselves. Again, we used Mechanical Turk, this time to re-assess some TREC topic/document pairs and thus see if we can "get rid of" relevance assessors by replacing them with a crowd working remotely on the Web. I'll discuss the preliminary results on the reliability of the crowd of assessors.
  
  (this is joint work with Omar Alonso, A9.com; thanks also to Dan Rose)
  

I'll publish the slides here, once ready. Meaning: after the talks :-)
S.