Readersourcing - some stuff

Just to collect some thoughts:

• The official version of my "Readersourcing - A Manifesto" paper is now available on publisher's website.
• There seems to be an increasing interest on issues related to peer review and scholarly publishing:


• M. Tamer Özsu writes on the ACM SIGMOD blog;
• Moshe Y. Vardi continues to write editorials on CACM: 1, 2, 3.
• Elsevier asks to the crowd for good ideas.
• I came across to this other website.

I admit that I still have to read everything but I have the feeling that:

• most people aim at a system evolution; Readersourcing is a revolution-oriente approach. Not sure which is wiser...
• Bits and pieces of the whole Readersourcing idea can be found here and there (e.g., in several of Elsevier's challenge ideas).

Future will tell...

S.

È tutto un magna magna

Citation cartel.

S.

make.tenure.FAST

:-)

S.

Readersourcing

Readersourcing.org. Have a look. (there you can also download the slides of my first invited talk)

S.

On peer review and scholarly publishing - again

Another interesting proposal, although my own papers are not cited ;-)

S.

Invited talks

I'll be giving two invited talks in the next months:

• Readersourcing: Scholarly publishing, peer review, and barefoot cobbler's children @ FDIA 2009 @ ESSIR 2009, 01/09/2009.
  
  Abstract:
  
  I will start from an introduction to the field of scholarly publishing, the main knowledge dissemination mechanism adopted by science, and I will pay particular attention to one of its most important aspects, peer review. I will present scholarly publishing and peer review aims and motivations, and discuss some of their limits: Nobel Prize winners experiencing rejected papers, fraudulent behavior, sometimes long publishing time, etc. I will then briefly mention Science 2.0, namely the use of Web 2.0 tools to do science in a hopefully more effective way.
  
  I will then move to the main aspect of the talk. My thesis is composed of three parts.
  
  (i) Peer review is a scarce resource, i.e., there are not enough good referees today. I will try to support this statement by something more solid than the usual anecdotal experience of being reject because of bad review(er)s --- that I'm sure almost any researcher has experienced.
  
  (ii) An alternative mechanism to peer review is available right out there, it is already widely used in the Web 2.0, it is quite a hot topic, and it probably is much studied and discussed by researchers: crowdsourcing. According to Web 2.0 enthusiasts, crowdsourcing allows to outsource to a large crowd tasks that are usually performed by a small group of experts. I think that peer review might be replaced --- or complemented --- by what we can name Readersourcing: a large crowd of readers that judge the papers that they read. Since most scholarly papers have many more readers than reviewers, this would allow to harness a large evaluation workforce. Today, readers's opinions usually are discussed very informally, have an impact on bibliographic citations and bibliometric indexes, or stay inside their own mind. In my opinion, it is quite curious that such an important resource, which is free, already available, used and studied by the research community in the Web 2.0 field, is not used at all in nowadays scholarly publishing, where the very same researchers publish their results.
  
  (iii) Of course, to get a wisdom of the crowd, some readers have to be more equal than others: expert readers should be more influential than naive readers. There are probably several possible choices to this aim; I suggest to use a mechanism that I proposed some years ago, and that allows to evaluate papers, authors, and readers in an objective way. I will close the talk by showing some preliminary experimental results that support this readersourcing proposal.
  
  Disclaimer: This talk might harm your career; don't blame me for that.
  
  P.S. Yes, this is somehow related to a previous post...
  
  
• Two Tales on Relevance Crowdsourcing: Criteria and Assessment @ GIScience Colloquium, University Zurich-Irchel, 13/10/2009.
  
  Abstract (DRAFT):
  
  In Information Retrieval (IR) and Web search, relevance is a central notion. I will discuss how to outsource to the crowd two relevance-related tasks. The first task concerns the elicitation of relevance criteria. After some results obtained in the 90es, relevance criteria (i.e., the features of the retrieved items that determine their relevance) seem well known and stable. We conjectured that for e-Commerce / product search the criteria might be different, and we used Amazon Mechanical Turk, a crowdsourcing platform, to find a confirmation of our hypothesis.
  
  The second task concerns effectiveness evaluation. A common evaluation methodology for search engines and IR systems is to rely on a benchmark (a.k.a. test collection); benchmarks need relevance assessment, i.e., to assess the relevance of documents to information needs; usually this task is done by experts, either paid for their work or participating in the evaluation exercise themselves. Again, we used Mechanical Turk, this time to re-assess some TREC topic/document pairs and thus see if we can "get rid of" relevance assessors by replacing them with a crowd working remotely on the Web. I'll discuss the preliminary results on the reliability of the crowd of assessors.
  
  (this is joint work with Omar Alonso, A9.com; thanks also to Dan Rose)
  

I'll publish the slides here, once ready. Meaning: after the talks :-)
S.

A good tutorial on publishing good research

A good tutorial on publishing good research (from William Webber’s Research Blog). I'd use it for my wanna-be-course.

S.

PhD course on Scientific information dissemination

I've been suggesting that PhD students at my department - and at my university as well - should be offered a seminar/course on "Scientific information dissemination" (I should find a better title, actually...). Topics:

• how to write a research paper
• how to give a good presentation
• tools for writing and presenting
• Web 2.0 as a scientific dissemination tool (blogs, wikis, youtube, facebook, etc.)

I also have half-volunteered to organize and teach such a seminar. Let's see...
S.

Research evaluation & bibliometrics

At my university there is a strong push, by... someone, to use bibliometric indicators to evaluate research productivity. My position is:

• Bibliometrics is useful. It would be crazy not to use it.
• Bibliometrics is not enough. It would be equally crazy to rely on bibliometrics only. A more general, scientometrics-based approach, should be used.
  
• Expert peer review, informed by bibliometrics but not only, is the only reliable evaluation mechanism.
• Evaluation needs resources (money, people, time, data, ...).
• Evaluation strategies and approaches should be decided by evaluation experts, not by beginners.
  

Probably, someone, somewhere in the world, is fighting trying to convince they colleagues and/or administrators, that bibliometrics is useful. At my university, I'm in the opposite position: I'm fighting trying to convince people that bibliometrics can give some indication, but it's just one of the parameters to be measured, and that it is stupid to rely on bibliometrics only.

BTW, so far I did not get much success - they just pretend I'm not saying anything. Well, actually they've changed from using bibliometrics to evaluate single researchers (a position that demonstrated their level of... erm let's be polite and say "expertise"...) to using bibliometrics to evaluate groups of researchers. But I'm stubborn ;-)

Anyway, I'm writing this post because I'm happy that in the last REF report one can read:

There was a strong consensus that bibliometrics are not sufficiently mature to be used formulaically or to replace expert review, but there is considerable scope for citation indicators to inform expert review in the REF.

Let's see if bibliometrics enthusiast at my university will take this into account, or just pretend it doesn't exist.

And, yes, of course, let's see if bibliometrics-retractors will take this into account, or just pretend it doesn't exist.

S.

If research papers...

... had a comment section...

S.

WikiTracer: Mapping the Wikisphere

• WikiTracer: Mapping the Wikisphere

  tags: wiki, ej

  • WikiTracer is a web service providing platform-independent analytics and comparative growth statistics for wikis.

Posted from Diigo.

S.

Dialogue on a Midspring's Night Dream in Dagstuhl

Prologue

[Any reference to real people is not casual. You'll recognize yourself if you were there. :) ]

N.B. So what's your proposal about?

K. Yes, describe it to us.

S. You can describe it in several ways. One...

K. Oh no, it can't work.

M. Sure it can't, readers will not behave properly.

N.B. Poor guy, he hasn't started yet. Wait a minute.

S. That's the usual reaction: have an opinion - a strong one - before even trying to understand my model. One: in current scholarly publishing system, the reviewers are the scarce resource.

K. I agree.

S. We are all reviewing more and more papers; reviewers are being paid; authors re-submit rejected papers without any change and (different) reviewers have to re-review them. And so on. The more the paper publication rate increases, the more the situation gets worst.

K.: Right. I know that. I've been working on that with Karl Popper, he was a good friend of mine. We wrote a paper on it, actually. But you should consider that if I ask people to review papers, they will do it.

S. You said that you agree that reviewers are the scarce resource.

K. I'll give you an example of that. Quite a good example actually. I just asked 150 top scientists to do some reviews, and all of them agreed. None refused.

S. Ok, sure, but we're talking about something different here.

N.B.: Guys, just look at your hands. You can see some body language there. But I don't understand what you're talking about.

S. - Ok, I'll show you some slides (picks up his laptop - a Mac, of course)

N.S. I'll go and grab some cheese. I'm hungry. They did not give us enough food for dinner.

A. I'm not hungry at all.

S. We're eating far too much.

A. I'll have some wine.

Everybody: Me too!

S. So; One: reviewers are a scarce resource. Two: there are a lot of readers out there.

D. Lots of what?

S. Readers. Scientists that read scientific papers, form an opinion about them, and take that opinion in their mind - or maybe share it with a few colleagues. The scientific community as a whole does not benefit from reader's knowledge. The scholarly publishing and knowledge dissemination field could do something similar to Web 2.0 by exploiting the "wisdom of readers": a scholarly paper is refereed by 2, 3, maybe 4 researchers, but it is hopefully read by dozens (10^1), hundreds (10^2), or even thousands (10^3) of researchers. And these researchers usually form an opinion about the paper. And this opinion is usually left inside their mind, or communicated in a very informal manner. We're just using the logarithm of the reading power that we have!

M. Wait wait, log of thousands is 3 not 4, you're cheating!!!

S. Sorry, ok, that's wrong, I had too much cheese... erm, beer, but you should get the whole idea anyway. It's log(x-1)...

M. You're cheating again! log(x-1) is undefined for no readers.

S. Come on... (lifting the middle finger of his right hand - and having some cheese)

D. Jesus these guys are crazy.

M. So what? Who cares? That's bullshit.

N.B. Are you suggesting to use readers judgments?

M. But you have to prove that your system is better than the h-index. The h-index measures researcher quality. I do have a high h-index.

S. Yes and no. Wait a minute and you'll see.

N.B. So you're going to suggest to use readers judgments. Oh no that can't work. I feel like a Strasbourg goose, but I'll have some more cheese anyway.

N.S. Yef cheefe if fey goot. Pfleafe, hafe fome. Gulp. But do you mean that if a student of mine judges my paper as crap I should be penalized for that?

S. To a certain extent yes. But, wait a minute, can I have some cheese?

D. - "Jesus, these guys are crazy."

S. We're eating far too much.

M. I'll go and skype my daughters. As we will see later, that'll be the last serious thing for today.

S. Readers judgments will be weighted on the basis of how good they are.

K. Right. But who decides if I am a good reader? I could, of course, but not anybody can. If you put a cat in a box and ask him to judge your papers, would you be happy? And what if the cat is dead? You know this is a known paradox in Quantum Mechanics, and it has actually been studyed by me and Zeilinger when he was a PhD student of mine. We proved that the cat paradox can be solved if you consider the probabilistic deontic paranormal logic of the unjustified true beliefs.

S. You are a good reader if you express good judgment.

K. Right. But ...

S. Stop stop stop! If you start again with that we won't get anything. So: the goodness of your judgment depends on the score of the paper after the last reader has read it. Of course you don't have the final paper score at that time; you can approximate it by using the current score. And this approximation will be revised as time goes on and new judgments on the same paper are expressed: this will cause the paper score to change, and the goodness of previous judgments to change as well, and so on. It's complex, but it's recursive, like PageRank, and this animation on my slides shows that...

K. (having some wine) So my judgment is good if it is close to the current paper score.

S. Yes and no.

K. Oh, we're playing quantum logic here. You know that in the 40es when a was a PhD student in Transilvania with Heisenberg we proved that, in quantum mechanics, professors can be old and young at the same time, provided that...

S. Stop stop stop!! Shut up! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!!

D. S., I'm wondering what kind of beer are you drinking?

S. It's not the beer, it the glass. This is the wrong glass for that kind of beer. You'd better drink it from the bottle. Like grappa.

N.S. Grappa? Who said grappa??

A. Friend, he said that, but there's no grappa here.

N.S. He's no more my friend.

M. I've just seen that my h-index is almost as K.'s.

N.B. I do feel like a Strasbourg goose now. I'll go and grab some cheese. And go to bed. It can't work. Democracy is not science.

S. I told you it's not democracy.

Epilogue of the prologue (i.e., after some cheese/wine/beer)

S. BTW, I even published a paper on it.

N. When? Where?

K. Why I don't know it?

S. 2003, On a peer reviewed journal. JASIST.

K. Ok, send me the paper. I'd love to read it.
K. Ok, send me the paper, so you stop saying stupid nonsense bullshit.

N.S. Yes send the paper.
N.S. You're no more my friend.

The cat in the box: meow!

A. Have you seen Copenhagen by M. Frayn?

D. These guys are crazy.

M.P. - Spam lovely spaam!

C. (just passing by) Guys, we're eating far too much!!

M.P. - Spam lovely spaam!

S. Stop stop stop!! Shut up! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!!


References:

• Monthy Python Spam video on YouTube; actually I prefer this version with Japanese (?) subtitles
• The paper (in a preprint version)
• A blog (discussing similar themes, and one more), and another post on another blog, and an interesting related project.

S.

What's wrong with scholarly publication

At the recent grappa ehm liquidpub workshop, I maintained the following position:

1) In today's scholarly publishing and knowledge dissemination field, peer review is the scarce resource. There is not enough reviewing force; reviews are often done quickly, if not badly; there is almost no acknowledgment for being a good peer reviewer; etc.

2) The scholarly publishing and knowledge dissemination field is not learning from what is being done for quality control out there on the Web: Web 2.0 (whatever it is) exploits the wisdom of crowd to attach ratings, opinions, tags, etc. to digital objects, apparently in an effective way. Think of digg, delicious, reddit, ebay, epinions, slashdot, karma, etc. etc.

3) The scholarly publishing and knowledge dissemination field could do something similar by exploiting the "wisdom of readers": a scholarly paper is refereed by 2, 3, maybe 4 researchers, but it is hopefully read by dozens (10^1), hundreds (10^2), or even thousands (10^3, 10^4) of researchers. And these researchers usually form an opinion about the paper. And this opinion is usually left and lost inside their mind, or communicated in a very informal manner. We, the scholars community, are just using the logarithm of the reading (and reviewing/rating/judging/evaluating) power that we have!

Of course there's the issue of distinguishing good and bad reviewers/readers. On that, time ago, I published a (peer reviewed!) paper about an alternative/supplementary concrete mechanism to peer review:

S. Mizzaro. Quality Control in Scholarly Publishing: A New Proposal, Journal of the American Society for Information Science and Technology, 54(11):989-1005, 2003. pdf (in a preprint version)

Also, recently we've been working at applying the same model to Wikipedia, where the rules are different from the scholarly publishing world: distributed authorship, dynamic/liquid papers, implicit judgments on the basis of the amount of editing, etc. We experimentally evaluated the approach, obtaining some encouraging preliminary results. That's a second paper:

Alberto Cusinato, Vincenzo Della Mea, Francesco Di Salvatore, Stefano Mizzaro. QuWi: Quality Control in Wikipedia. In WICOW 2009: 3rd Workshop on Information Credibility on the Web in conjunction with 18th World Wide Web, Madrid, 20 April 2009, forthcoming.

Time will tell if this model will be used, if it would work in the real world, etc. But for sure there is some stir about the scholarly publishing:

• A blog (discussing similar themes, and one more), and another post on another blog (Update: another one).
• Three interesting pieces.
  
• Some interesting web pages.

S.