Wednesday 25 March 2009

Dialogue on a Midspring's Night Dream in Dagstuhl

Prologue

[Any reference to real people is not casual. You'll recognize yourself if you were there. :) ]

N.B. So what's your proposal about?

K. Yes, describe it to us.

S. You can describe it in several ways. One...

K. Oh no, it can't work.

M. Sure it can't, readers will not behave properly.

N.B. Poor guy, he hasn't started yet. Wait a minute.

S. That's the usual reaction: have an opinion - a strong one - before even trying to understand my model. One: in current scholarly publishing system, the reviewers are the scarce resource.

K. I agree.

S. We are all reviewing more and more papers; reviewers are being paid; authors re-submit rejected papers without any change and (different) reviewers have to re-review them. And so on. The more the paper publication rate increases, the more the situation gets worst.

K.: Right. I know that. I've been working on that with Karl Popper, he was a good friend of mine. We wrote a paper on it, actually. But you should consider that if I ask people to review papers, they will do it.

S. You said that you agree that reviewers are the scarce resource.

K. I'll give you an example of that. Quite a good example actually. I just asked 150 top scientists to do some reviews, and all of them agreed. None refused.

S. Ok, sure, but we're talking about something different here.

N.B.: Guys, just look at your hands. You can see some body language there. But I don't understand what you're talking about.

S. - Ok, I'll show you some slides (picks up his laptop - a Mac, of course)

N.S. I'll go and grab some cheese. I'm hungry. They did not give us enough food for dinner.

A. I'm not hungry at all.

S. We're eating far too much.

A. I'll have some wine.

Everybody: Me too!

S. So; One: reviewers are a scarce resource. Two: there are a lot of readers out there.

D. Lots of what?

S. Readers. Scientists that read scientific papers, form an opinion about them, and take that opinion in their mind - or maybe share it with a few colleagues. The scientific community as a whole does not benefit from reader's knowledge. The scholarly publishing and knowledge dissemination field could do something similar to Web 2.0 by exploiting the "wisdom of readers": a scholarly paper is refereed by 2, 3, maybe 4 researchers, but it is hopefully read by dozens (10^1), hundreds (10^2), or even thousands (10^3) of researchers. And these researchers usually form an opinion about the paper. And this opinion is usually left inside their mind, or communicated in a very informal manner. We're just using the logarithm of the reading power that we have!

M. Wait wait, log of thousands is 3 not 4, you're cheating!!!

S. Sorry, ok, that's wrong, I had too much cheese... erm, beer, but you should get the whole idea anyway. It's log(x-1)...

M. You're cheating again! log(x-1) is undefined for no readers.

S. Come on... (lifting the middle finger of his right hand - and having some cheese)

D. Jesus these guys are crazy.

M. So what? Who cares? That's bullshit.

N.B. Are you suggesting to use readers judgments?

M. But you have to prove that your system is better than the h-index. The h-index measures researcher quality. I do have a high h-index.

S. Yes and no. Wait a minute and you'll see.

N.B. So you're going to suggest to use readers judgments. Oh no that can't work. I feel like a Strasbourg goose, but I'll have some more cheese anyway.

N.S. Yef cheefe if fey goot. Pfleafe, hafe fome. Gulp. But do you mean that if a student of mine judges my paper as crap I should be penalized for that?

S. To a certain extent yes. But, wait a minute, can I have some cheese?

D. - "Jesus, these guys are crazy."

S. We're eating far too much.

M. I'll go and skype my daughters. As we will see later, that'll be the last serious thing for today.

S. Readers judgments will be weighted on the basis of how good they are.

K. Right. But who decides if I am a good reader? I could, of course, but not anybody can. If you put a cat in a box and ask him to judge your papers, would you be happy? And what if the cat is dead? You know this is a known paradox in Quantum Mechanics, and it has actually been studyed by me and Zeilinger when he was a PhD student of mine. We proved that the cat paradox can be solved if you consider the probabilistic deontic paranormal logic of the unjustified true beliefs.

S. You are a good reader if you express good judgment.

K. Right. But ...

S. Stop stop stop! If you start again with that we won't get anything. So: the goodness of your judgment depends on the score of the paper after the last reader has read it. Of course you don't have the final paper score at that time; you can approximate it by using the current score. And this approximation will be revised as time goes on and new judgments on the same paper are expressed: this will cause the paper score to change, and the goodness of previous judgments to change as well, and so on. It's complex, but it's recursive, like PageRank, and this animation on my slides shows that...

K. (having some wine) So my judgment is good if it is close to the current paper score.

S. Yes and no.

K. Oh, we're playing quantum logic here. You know that in the 40es when a was a PhD student in Transilvania with Heisenberg we proved that, in quantum mechanics, professors can be old and young at the same time, provided that...

S. Stop stop stop!! Shut up! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!!

D. S., I'm wondering what kind of beer are you drinking?

S. It's not the beer, it the glass. This is the wrong glass for that kind of beer. You'd better drink it from the bottle. Like grappa.

N.S. Grappa? Who said grappa??

A. Friend, he said that, but there's no grappa here.

N.S. He's no more my friend.

M. I've just seen that my h-index is almost as K.'s.

N.B. I do feel like a Strasbourg goose now. I'll go and grab some cheese. And go to bed. It can't work. Democracy is not science.

S. I told you it's not democracy.

Epilogue of the prologue (i.e., after some cheese/wine/beer)

S. BTW, I even published a paper on it.

N. When? Where?

K. Why I don't know it?

S. 2003, On a peer reviewed journal. JASIST.

K. Ok, send me the paper. I'd love to read it.
K. Ok, send me the paper, so you stop saying stupid nonsense bullshit.

N.S. Yes send the paper.
N.S. You're no more my friend.

The cat in the box: meow!

A. Have you seen Copenhagen by M. Frayn?

D. These guys are crazy.

M.P. - Spam lovely spaam!

C. (just passing by) Guys, we're eating far too much!!

M.P. - Spam lovely spaam!

S. Stop stop stop!! Shut up! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!!


References:

S.