Computer science education

Un post interessante...

S.

"No flashy. This is Science." [censored] -- Or: How two jetlagged IR researchers met and had an idea

[Prologue: the beginning of a beautiful friendship]

J. Hi. Can I sit here?
S. Sure. Hi. I don't think we met before. Nice to meet you, I'm S.
J. I'm J.
S. Nice place here, isn't it?
J. Yep! Beach this afternoon?
S. Sure!
J. Ok, let's pretend we do some work first. So, what are you working on?
S. Oh, several things, blablabla [snip] And you?
J. Well, I did blablabla... and blabla... and I also published some papers on formal analysis of clustering metrics.
S. Interesting. I also started something similar for IR metrics years ago, but I never managed to publish it.
J. Really?
S. Yep. I'll show you [opening his laptop]. See, I defined a framework based on measurement theory, then I defined some axioms...
J. That's crazy! I did exactly the same!
S. ... then some desiderata...
J. Exactly the same!
S. ... Yes but you published, I didn't...
J. ... because you tried to do everything, look here, that's crazy!
S. Yes, I'm a bit, sure... and then blablabla...
J. blablabla [very technical details here - ok, ok, I admit I do not remember that!]
S. blablabla [very technical details here - ok, ok, I admit I do not remember that!]
J. Well, why not proposing this as an idea for SWIRL research directions?
S. We might indeed!
J. Ok, beach time now.
S. Beach!

[Chapter 1: split groups doing Science]

M. So, let's have a round of the table and everyone presents his own idea.
[various good ideas...everyone presents just one, accordingly to the rules. But you know that Italians and Rules do not fit well in the same sentence, so...]
S. I do have two ideas. The first is not very exciting, but it's Science. The second is more exciting...
Others. Well, tell us both. [meaning: the usual Italian breaking rules...]
S. The first is about IR effectiveness metrics. We have around 100 metrics, counting the system-oriented ones only. The research project would be to find formal properties that each metric should satisfy.
Others. So do you mean...?
S. To define axioms that metrics should satisfy. For instance, first retrieved documents should weight more (or not less) than following docs, etc.
Others. Hm. What's that for?
S. Oh well, for instance we could understand why nDCG discount function is defined in that way...
Others. Cool. We need a name...
A. Axiometrics!
Everyone. That's great!!

[Chapter 2: Axiometrics]

S. Hi J.. Did you mention the idea about metrics in your group?
J. No, I didn't, and you?
S. Yes, I did, and guess it, it was selected!
J. Really?? Great!!
S. And A. invented a cool name: Axiometrics
J. That's wonderful!
S. Let's have some coffe, I'm still jetlagged.
J. Yes. And beach later. And beers.

[Chapter 3: Coffee and beers]

[after some hours, and coffee. And beach. And beers.]
S., J., A. M. N. Ok, let's write this short report about Axiometrics in SWIRL...
... well, let's have a short and last beach session first!


[Chapter 4: Planes, emails, and deadlines]


[After some long flights back home, tons of unanswered emails and student requests, expenses claim forms, etc. etc., and just a few spare days before the deadline...]
S. J., do you think it is reasonable to submit a Google research grant proposal on Axiometrics? Or is it a stupid idea?
J. That's a great idea!
S. Let's involve E. as well, he is interested.
[some hard science follows: bibliography is polished, CVs are created, margins and fonts are modified, PDFs generated...]


[Chapter 5: 4th of July]
Guys, we won!!

P.S. Thanks to Julio, Evangelos, Arjen, Marteen, Nicola. And to SWIRL and Mark!


S.

SWIRL 2012 - 2

I forgot two things:

1) the tagcloud of the nominated papers. I simply cut&pasted the text at http://www.cs.rmit.edu.au/swirl12/discussion.php  into Worlde and I got:

(I think it is clear that "evaluation" is a hot issue, once removed obvious terms like "paper", "information", "retrieval", etc., but some might disagree)

2) I believe there was an elephant in the room: Crowdsourcing.

S.

SWIRL 2012

Last week I attended SWIRL 2012: these are my highlights.

Facts.

SWIRL 2012 has been a workshop where around 50 jetlagged top IR researchers gathered together essentially to experience summer during winter and incidentally to discuss the future of IR research. I am not sure that I count among the top IR researchers. Actually, probably the selection mechanism was more oriented towards the most crazy IR researchers since some really top were missing and some very crazy were definitely there. So I am not sure that I count among the top IR researchers, we can discuss if I count among the top crazy ones, but I'm sure I count among the top jetlagged ones. Luckily enough, all the others were jetlagged as well, and looked like Australians in Europe, so I managed to camouflage myself somehow. Also, and surprisingly, being very jetlagged even helped, since we had long discussions during the nights, and we managed to use the nights to read our emails :) and have the days free for working.

The workshop started with the usual nice dinner in a nice hotel. Well, actually we had a prologue, with some talks at RMIT from some IR researchers. They were, guess, extremely jetlagged, so probably not all the talks were so good, and yes I was one of the speakers. You see what I mean? Anyway, I presented Readersourcing, and I learned two things:

1. Now I know who was one of the referees of the famous Go*gle/PageRank paper rejected from SIGIR; and that that submitted version wasn't that good... Anyway...
2. When presenting Readersourcing in these years, I managed to discuss... ehm let's say to argue with S, K, and B. I did not record the first "discussion", but I did for the second one here. Now, the point is: shouldn't I seriously consider to quit this line of research?!?

Anyway, after that, the workshop started with the usual nice dinner in a nice hotel. During it I learned three things:

1. Evaluation? We haven't had enough.
2. How to wreck a nice beach (don't ask me to explain this, ask David).
3. You can live your life as a party (ask Leif)

Then the crowd of 50-top-and-seriously-jetlagged-IR-researchers was packed into a bus and sent to Lorne to experience real summer. Indeed, the first thing we did in Lorne was to go to the nice beach (...). But after that we started to do some serious work. I don't recall all the details, probably because I was considerably jetlagged, but I do remember the main things. We were divided into 6 random groups that had to produce some ideas about the future of IR, i.e., research topics and directions that we felt important for the field. After that, we pooled all the ideas together (you know, we like pooling; actually, it is surprising to me that we didn't manage to stick into this process a logarithm, or a significance test, or the definition of a novel evaluation metric) and we voted the 6 top ideas. Indeed we didn't like the voting result too much so we slightly modified it (I'd say "the Italian way", but strangely there were no Italians driving this process) by merging, splitting, adding, removing, changing, deleting, twinkling, etc., and we obtained The-list-of-the-6-top-ideas-that-will-change-the-IR-world-in-the-future:

1. Mobile IR
2. Structured and unstructured information
3. users People
4. SmartIR (aka information literacy)
5. "(less than) zero query" [sic]
6. and, of course, Argy-Bargy

The other ideas were not killed, and will go back into play later on. We were then asked to form six groups, each one aiming at discussing one of the six ideas. Looking at the list it was very difficult for me to choose which group to join. Obviously, any schoolboy will understand the foundational and metaphysics nature of Argy-Bargy. Anyone will agree that we shall have no other information besides structured and unstructured. We can't kill users People, of course (fair enough, we didn't write "students"). SmartIR is the smarter term in the list. I've been working for the last 10 years, and published most of my last papers, on "(less than) zero query" [sic], although with different, and equally stupid strange labels like "query-free IR", "zero-interaction interactive IR", "browsing the virtual space by walking in the physical space", etc.

But at the end I joined the Mobile IR group. And, guess it, it was not about Mobile IR. It was about Understanding what Mobile IR is. Funny: a group that did not understand what it was about but aimed at understanding what its (mistaken) topic was about. How couldn't we succeed? Plus, when people chose their own group, strange things happened. Some people were not able to find their group; perhaps it was on the nice beach? Some people (those above plus others) joined another "wrong" group, and some of them, just to avoid being idle, decided to act as trolls. We had two trolls in the Mobile IR group, and you will understand how that made the discussion far more interesting, lively, polite, and constructive than you could ever imagine. Luckily, the night came, and after a good sleep (or perhaps some good chats, beers, barbecues, etc., since we were seriously jetlagged and can't manage to have that much sleep anyway), the morning after, Jamie and Vanessa had a clear vision of what the group was about and started drafting the report (each group was meant to write a report). Not having trolls around early morning probably helped. Did trolls suffer from jetlag? Did trolls take surfer lessons? We will never know.

As mentioned above, the other not-top-6-best-great-ideas were not killed, and some still-quite-heavily-jetlagged-IR-researchers volunteered to write a short report on those. Now, I have to mention the best name of the workshop: Axiometrics (credits to Arjen), a research line aimed at defining axioms in order to understand the about 100 IR effectiveness metrics. BTW, we might think of Anatometrics as well.

Comments?

So the workshop was great, the organizers incredibly managed to obtain something out of about-50-top-crazy-IR-researchers-that-as-you-know-were-very-jetlagged, the discussions were interesting, I managed to have some good ideas and contacts for future paper writing, and perhaps some contact for my sabbatical as well (I'm looking for places were to stay during my sabbatical; please let me know if you're interested. I promise that I won't be so jetlagged for the whole sabbatical duration.) Plus, I've been out of business recently, for several reasons including lack of funds, A.'s birth, etc., and it was really nice to meet some old good friends and some new ones.

Criticisms? I always have. We could have made use of some "Social Web/Web2.0" tools, like Gdocs, Facebook, Twitter, to have a virtual discussion as well and, for example, to vote the ideas (did I hint that the voting mechanism was a bit... "italianized"? Now, we all know about Arrow's theorem, but that was far beyond that). Someone said that s/he had the impression that we were simply drafting a report to make easier for US and Australian researchers to get funds. Someone had the impression that the meeting was a bit too "old fashioned". But as I wrote, the organizers did manage to get something out of about-50-top-IR-researchers-that-as-you-know-were-very-jetlagged, and this is an enormous success.

Quotations!

Some interesting sentences were uttered during those days, and shall never be forgotten:

• Cloud computing is not transparent.
• You can't leave indoor outdoor, but you can't leave outdoor outdoor either.
• How to wreck a nice beach.
• Life as a party.
• (Julio please help with the other one)
• (anybody welcome to add)

Post workshop!!

Once back in Melbourne, a (randomly) selected subgroup of all those (somehow) 50 selected IR researcher had an interesting post-workshop, post-dinner, during-beer mini-workshop on p*orn and user models. I took some pictures of the participants:

As you can see, we range from someone (pretending to be?) not interested, or perhaps simply jetlagged, to someone really having fun, to a shy guy who doesn't want to be recognized, to someone counting beers (not an easy task!), to someone hiding in the shade. I learned some interesting statistics about user features. Anyway, guys, it was fun. Thanks!

S.

Post, P.S. e commenti

A volte il P.S.

Ps. A seguito del mio post su Valutazione e burocrazia mi è stato segnalato che, secondo sviluppi recenti di cui non ero a conoscenza, è stato deciso che nella prossima tornata del Rae non si farà uso di un sistema automatico di valutazione.

e i commenti sono più importanti dei post.

S.

Premio Telecom

Notizia :)

20 mila euro per un programma sviluppato al dipartimento di Matematica e informatica

Ricerca e web, premio Telecom a un progetto nato nei laboratori dell'ateneo friulano

Sistema automatico per ottenere da internet le informazioni giuste, nel posto giusto, al momento giusto

Un programma che seleziona automaticamente da internet informazioni e applicazioni utilizzabili dall’utente di un dispositivo mobile (palmari e smarth phone) sulla base del luogo e della situazione in cui si trova. È il Context-Aware Browser, il sistema ideato da un gruppo di informatici del laboratorio di Sistemi mobili dipendenti dal contesto (Smdc) dell’università di Udine, premiato da Telecom Italia con un riconoscimento di 20 mila euro nell’ambito del progetto Working Capital. Scopo dell’iniziativa, infatti, è sostenere l’innovazione attraverso la valorizzazione dei giovani talenti e la promozione delle iniziative imprenditoriali in internet.

Il Context-Aware Browser è un browser (programma che consente di visualizzare i contenuti delle pagine web e di interagire con essi) che permette una navigazione nel mondo digitale sulla base della situazione in cui ci si trova nel mondo reale. «L’idea generale alla base del progetto – spiega il portavoce del gruppo, l’udinese Luca Vassena, dottorando in Informatica – è poter ottenere automaticamente le applicazioni e le informazioni giuste, nel posto giusto, al momento giusto».

Ad esempio, entrando in una città il Context-Aware Browser mostra automaticamente all’utente informazioni relative alla città, ai luoghi d’interesse, agli eventi ecc. Oppure, all’ora di cena, se l’utente non è a casa, il programma consiglia automaticamente i locali dove poter andare a mangiare, filtrandoli in base alle sue preferenze. Quando poi una persona entra nella propria abitazione, il dispositivo mobile fornisce automaticamente l’applicazione web che si collega con l’impianto domotico per controllare la casa. L’applicazione viene quindi rimossa nel momento in cui la persona esce dall’abitazione.

Nel centro commerciale, invece, il dispositivo ottiene l’applicazione web per gestire la lista della spesa e permette di ricevere avvisi pubblicitari mirati in base al singolo utente e alle sue preferenze. E ancora, in un museo il dispositivo dell’utente mostra automaticamente la guida turistica relativa a quel museo, dando informazioni dettagliate sulle opere a cui il visitatore si avvicina o consigliando percorsi particolari in base al suo profilo.

Al progetto, realizzato presso il laboratorio di Sistemi mobili dipendenti dal contesto del dipartimento di Matematica e informatica, in collaborazione con lo spin off MoBe srl, hanno lavorato un gruppo di ricercatori e assegnisti di ricerca coordinati dai docenti Paolo Coppola e Stefano Mizzaro.
25/10/2010

S.

A Classification of Computer Science Research :-)

A Classification of Computer Science Research (from here):

This is a view of computer science research. Sarcasm abounds. This is supposed to be funny, but it can offend people. Don't read if you are easily offended, and don't get angry if your most favorite research topic is not presented appropriately. If your most favorite pet is not here, please let me know.

Security: The art of thinking about unsolvable problems.

Theory: The art of creating unsolvable problems.

Systems: The continuous re-implementation of ideas that appeared in 1950's.

Compilers: The science of arguing that every thing is either NP-complete or undecidable, and showing that neither is relevant.

Artificial Intelligence: A line of research whose existence is motivated and justified by the failure of natural intelligence, or the lack thereof.

Networking: An excuse for doing research in security.

Computer Architecture: The only successful branch of computer science research, although it has nothing to do with science, or research.

Graphics: Drawing pots and kettles on computer screens.

Databases: A research topic that was resolved in the 60's.

Parallel Processing: This is what you claim to be doing when you want the government to spend a lot of money to support you.

HCI: A philosopher in front of a Mac.

Geometric Modelling: That's what people who get tired of theory move to. They then retire.

Modelling and Simulation: A glorified priority queue that has been hacked to death.

Programming Languages() = Incomprehensible Math + Programming Langauges();

Software Engineering: Creating upper layers of software, then moving the bugs from the lower layers to the newly created layers. Repeat until there are no bugs or you die.

Fault Tolerance: A line of research that considers paranoia a science.

Citazione

“La tecnologia si sta affrancando dalla scienza e il mare che da sempre divide il dire dal fare oggi si naviga all’incontrario, nel senso che è molto più semplice fare che dire, cioè capire e spiegare” 

(Giuseppe O. Longo, Homo Technologicus, Meltemi, 2001)

S.

PhD in Computer Science @ Udine

We have a PhD degree in Computer Science at the University of Udine, Dept. of Mathematics and Computer Science. If you're interested, you have only two days left to submit an application (and you can do that online).

S.

Readersourcing

Readersourcing.org. Have a look. (there you can also download the slides of my first invited talk)

S.

Invited talks

I'll be giving two invited talks in the next months:

• Readersourcing: Scholarly publishing, peer review, and barefoot cobbler's children @ FDIA 2009 @ ESSIR 2009, 01/09/2009.
  
  Abstract:
  
  I will start from an introduction to the field of scholarly publishing, the main knowledge dissemination mechanism adopted by science, and I will pay particular attention to one of its most important aspects, peer review. I will present scholarly publishing and peer review aims and motivations, and discuss some of their limits: Nobel Prize winners experiencing rejected papers, fraudulent behavior, sometimes long publishing time, etc. I will then briefly mention Science 2.0, namely the use of Web 2.0 tools to do science in a hopefully more effective way.
  
  I will then move to the main aspect of the talk. My thesis is composed of three parts.
  
  (i) Peer review is a scarce resource, i.e., there are not enough good referees today. I will try to support this statement by something more solid than the usual anecdotal experience of being reject because of bad review(er)s --- that I'm sure almost any researcher has experienced.
  
  (ii) An alternative mechanism to peer review is available right out there, it is already widely used in the Web 2.0, it is quite a hot topic, and it probably is much studied and discussed by researchers: crowdsourcing. According to Web 2.0 enthusiasts, crowdsourcing allows to outsource to a large crowd tasks that are usually performed by a small group of experts. I think that peer review might be replaced --- or complemented --- by what we can name Readersourcing: a large crowd of readers that judge the papers that they read. Since most scholarly papers have many more readers than reviewers, this would allow to harness a large evaluation workforce. Today, readers's opinions usually are discussed very informally, have an impact on bibliographic citations and bibliometric indexes, or stay inside their own mind. In my opinion, it is quite curious that such an important resource, which is free, already available, used and studied by the research community in the Web 2.0 field, is not used at all in nowadays scholarly publishing, where the very same researchers publish their results.
  
  (iii) Of course, to get a wisdom of the crowd, some readers have to be more equal than others: expert readers should be more influential than naive readers. There are probably several possible choices to this aim; I suggest to use a mechanism that I proposed some years ago, and that allows to evaluate papers, authors, and readers in an objective way. I will close the talk by showing some preliminary experimental results that support this readersourcing proposal.
  
  Disclaimer: This talk might harm your career; don't blame me for that.
  
  P.S. Yes, this is somehow related to a previous post...
  
  
• Two Tales on Relevance Crowdsourcing: Criteria and Assessment @ GIScience Colloquium, University Zurich-Irchel, 13/10/2009.
  
  Abstract (DRAFT):
  
  In Information Retrieval (IR) and Web search, relevance is a central notion. I will discuss how to outsource to the crowd two relevance-related tasks. The first task concerns the elicitation of relevance criteria. After some results obtained in the 90es, relevance criteria (i.e., the features of the retrieved items that determine their relevance) seem well known and stable. We conjectured that for e-Commerce / product search the criteria might be different, and we used Amazon Mechanical Turk, a crowdsourcing platform, to find a confirmation of our hypothesis.
  
  The second task concerns effectiveness evaluation. A common evaluation methodology for search engines and IR systems is to rely on a benchmark (a.k.a. test collection); benchmarks need relevance assessment, i.e., to assess the relevance of documents to information needs; usually this task is done by experts, either paid for their work or participating in the evaluation exercise themselves. Again, we used Mechanical Turk, this time to re-assess some TREC topic/document pairs and thus see if we can "get rid of" relevance assessors by replacing them with a crowd working remotely on the Web. I'll discuss the preliminary results on the reliability of the crowd of assessors.
  
  (this is joint work with Omar Alonso, A9.com; thanks also to Dan Rose)
  

I'll publish the slides here, once ready. Meaning: after the talks :-)
S.

A good tutorial on publishing good research

A good tutorial on publishing good research (from William Webber’s Research Blog). I'd use it for my wanna-be-course.

S.

Questa poi...

:-)

S.

PhD course on Scientific information dissemination

I've been suggesting that PhD students at my department - and at my university as well - should be offered a seminar/course on "Scientific information dissemination" (I should find a better title, actually...). Topics:

• how to write a research paper
• how to give a good presentation
• tools for writing and presenting
• Web 2.0 as a scientific dissemination tool (blogs, wikis, youtube, facebook, etc.)

I also have half-volunteered to organize and teach such a seminar. Let's see...
S.

Research evaluation & bibliometrics

At my university there is a strong push, by... someone, to use bibliometric indicators to evaluate research productivity. My position is:

• Bibliometrics is useful. It would be crazy not to use it.
• Bibliometrics is not enough. It would be equally crazy to rely on bibliometrics only. A more general, scientometrics-based approach, should be used.
  
• Expert peer review, informed by bibliometrics but not only, is the only reliable evaluation mechanism.
• Evaluation needs resources (money, people, time, data, ...).
• Evaluation strategies and approaches should be decided by evaluation experts, not by beginners.
  

Probably, someone, somewhere in the world, is fighting trying to convince they colleagues and/or administrators, that bibliometrics is useful. At my university, I'm in the opposite position: I'm fighting trying to convince people that bibliometrics can give some indication, but it's just one of the parameters to be measured, and that it is stupid to rely on bibliometrics only.

BTW, so far I did not get much success - they just pretend I'm not saying anything. Well, actually they've changed from using bibliometrics to evaluate single researchers (a position that demonstrated their level of... erm let's be polite and say "expertise"...) to using bibliometrics to evaluate groups of researchers. But I'm stubborn ;-)

Anyway, I'm writing this post because I'm happy that in the last REF report one can read:

There was a strong consensus that bibliometrics are not sufficiently mature to be used formulaically or to replace expert review, but there is considerable scope for citation indicators to inform expert review in the REF.

Let's see if bibliometrics enthusiast at my university will take this into account, or just pretend it doesn't exist.

And, yes, of course, let's see if bibliometrics-retractors will take this into account, or just pretend it doesn't exist.

S.

If research papers...

... had a comment section...

S.

WikiTracer: Mapping the Wikisphere

• WikiTracer: Mapping the Wikisphere

  tags: wiki, ej

  • WikiTracer is a web service providing platform-independent analytics and comparative growth statistics for wikis.

Posted from Diigo.

S.

Dialogue on a Midspring's Night Dream in Dagstuhl

Prologue

[Any reference to real people is not casual. You'll recognize yourself if you were there. :) ]

N.B. So what's your proposal about?

K. Yes, describe it to us.

S. You can describe it in several ways. One...

K. Oh no, it can't work.

M. Sure it can't, readers will not behave properly.

N.B. Poor guy, he hasn't started yet. Wait a minute.

S. That's the usual reaction: have an opinion - a strong one - before even trying to understand my model. One: in current scholarly publishing system, the reviewers are the scarce resource.

K. I agree.

S. We are all reviewing more and more papers; reviewers are being paid; authors re-submit rejected papers without any change and (different) reviewers have to re-review them. And so on. The more the paper publication rate increases, the more the situation gets worst.

K.: Right. I know that. I've been working on that with Karl Popper, he was a good friend of mine. We wrote a paper on it, actually. But you should consider that if I ask people to review papers, they will do it.

S. You said that you agree that reviewers are the scarce resource.

K. I'll give you an example of that. Quite a good example actually. I just asked 150 top scientists to do some reviews, and all of them agreed. None refused.

S. Ok, sure, but we're talking about something different here.

N.B.: Guys, just look at your hands. You can see some body language there. But I don't understand what you're talking about.

S. - Ok, I'll show you some slides (picks up his laptop - a Mac, of course)

N.S. I'll go and grab some cheese. I'm hungry. They did not give us enough food for dinner.

A. I'm not hungry at all.

S. We're eating far too much.

A. I'll have some wine.

Everybody: Me too!

S. So; One: reviewers are a scarce resource. Two: there are a lot of readers out there.

D. Lots of what?

S. Readers. Scientists that read scientific papers, form an opinion about them, and take that opinion in their mind - or maybe share it with a few colleagues. The scientific community as a whole does not benefit from reader's knowledge. The scholarly publishing and knowledge dissemination field could do something similar to Web 2.0 by exploiting the "wisdom of readers": a scholarly paper is refereed by 2, 3, maybe 4 researchers, but it is hopefully read by dozens (10^1), hundreds (10^2), or even thousands (10^3) of researchers. And these researchers usually form an opinion about the paper. And this opinion is usually left inside their mind, or communicated in a very informal manner. We're just using the logarithm of the reading power that we have!

M. Wait wait, log of thousands is 3 not 4, you're cheating!!!

S. Sorry, ok, that's wrong, I had too much cheese... erm, beer, but you should get the whole idea anyway. It's log(x-1)...

M. You're cheating again! log(x-1) is undefined for no readers.

S. Come on... (lifting the middle finger of his right hand - and having some cheese)

D. Jesus these guys are crazy.

M. So what? Who cares? That's bullshit.

N.B. Are you suggesting to use readers judgments?

M. But you have to prove that your system is better than the h-index. The h-index measures researcher quality. I do have a high h-index.

S. Yes and no. Wait a minute and you'll see.

N.B. So you're going to suggest to use readers judgments. Oh no that can't work. I feel like a Strasbourg goose, but I'll have some more cheese anyway.

N.S. Yef cheefe if fey goot. Pfleafe, hafe fome. Gulp. But do you mean that if a student of mine judges my paper as crap I should be penalized for that?

S. To a certain extent yes. But, wait a minute, can I have some cheese?

D. - "Jesus, these guys are crazy."

S. We're eating far too much.

M. I'll go and skype my daughters. As we will see later, that'll be the last serious thing for today.

S. Readers judgments will be weighted on the basis of how good they are.

K. Right. But who decides if I am a good reader? I could, of course, but not anybody can. If you put a cat in a box and ask him to judge your papers, would you be happy? And what if the cat is dead? You know this is a known paradox in Quantum Mechanics, and it has actually been studyed by me and Zeilinger when he was a PhD student of mine. We proved that the cat paradox can be solved if you consider the probabilistic deontic paranormal logic of the unjustified true beliefs.

S. You are a good reader if you express good judgment.

K. Right. But ...

S. Stop stop stop! If you start again with that we won't get anything. So: the goodness of your judgment depends on the score of the paper after the last reader has read it. Of course you don't have the final paper score at that time; you can approximate it by using the current score. And this approximation will be revised as time goes on and new judgments on the same paper are expressed: this will cause the paper score to change, and the goodness of previous judgments to change as well, and so on. It's complex, but it's recursive, like PageRank, and this animation on my slides shows that...

K. (having some wine) So my judgment is good if it is close to the current paper score.

S. Yes and no.

K. Oh, we're playing quantum logic here. You know that in the 40es when a was a PhD student in Transilvania with Heisenberg we proved that, in quantum mechanics, professors can be old and young at the same time, provided that...

S. Stop stop stop!! Shut up! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!!

D. S., I'm wondering what kind of beer are you drinking?

S. It's not the beer, it the glass. This is the wrong glass for that kind of beer. You'd better drink it from the bottle. Like grappa.

N.S. Grappa? Who said grappa??

A. Friend, he said that, but there's no grappa here.

N.S. He's no more my friend.

M. I've just seen that my h-index is almost as K.'s.

N.B. I do feel like a Strasbourg goose now. I'll go and grab some cheese. And go to bed. It can't work. Democracy is not science.

S. I told you it's not democracy.

Epilogue of the prologue (i.e., after some cheese/wine/beer)

S. BTW, I even published a paper on it.

N. When? Where?

K. Why I don't know it?

S. 2003, On a peer reviewed journal. JASIST.

K. Ok, send me the paper. I'd love to read it.
K. Ok, send me the paper, so you stop saying stupid nonsense bullshit.

N.S. Yes send the paper.
N.S. You're no more my friend.

The cat in the box: meow!

A. Have you seen Copenhagen by M. Frayn?

D. These guys are crazy.

M.P. - Spam lovely spaam!

C. (just passing by) Guys, we're eating far too much!!

M.P. - Spam lovely spaam!

S. Stop stop stop!! Shut up! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!! Shuuuuut uuuuuppppp!!


References:

• Monthy Python Spam video on YouTube; actually I prefer this version with Japanese (?) subtitles
• The paper (in a preprint version)
• A blog (discussing similar themes, and one more), and another post on another blog, and an interesting related project.

S.

What's wrong with scholarly publication

At the recent grappa ehm liquidpub workshop, I maintained the following position:

1) In today's scholarly publishing and knowledge dissemination field, peer review is the scarce resource. There is not enough reviewing force; reviews are often done quickly, if not badly; there is almost no acknowledgment for being a good peer reviewer; etc.

2) The scholarly publishing and knowledge dissemination field is not learning from what is being done for quality control out there on the Web: Web 2.0 (whatever it is) exploits the wisdom of crowd to attach ratings, opinions, tags, etc. to digital objects, apparently in an effective way. Think of digg, delicious, reddit, ebay, epinions, slashdot, karma, etc. etc.

3) The scholarly publishing and knowledge dissemination field could do something similar by exploiting the "wisdom of readers": a scholarly paper is refereed by 2, 3, maybe 4 researchers, but it is hopefully read by dozens (10^1), hundreds (10^2), or even thousands (10^3, 10^4) of researchers. And these researchers usually form an opinion about the paper. And this opinion is usually left and lost inside their mind, or communicated in a very informal manner. We, the scholars community, are just using the logarithm of the reading (and reviewing/rating/judging/evaluating) power that we have!

Of course there's the issue of distinguishing good and bad reviewers/readers. On that, time ago, I published a (peer reviewed!) paper about an alternative/supplementary concrete mechanism to peer review:

S. Mizzaro. Quality Control in Scholarly Publishing: A New Proposal, Journal of the American Society for Information Science and Technology, 54(11):989-1005, 2003. pdf (in a preprint version)

Also, recently we've been working at applying the same model to Wikipedia, where the rules are different from the scholarly publishing world: distributed authorship, dynamic/liquid papers, implicit judgments on the basis of the amount of editing, etc. We experimentally evaluated the approach, obtaining some encouraging preliminary results. That's a second paper:

Alberto Cusinato, Vincenzo Della Mea, Francesco Di Salvatore, Stefano Mizzaro. QuWi: Quality Control in Wikipedia. In WICOW 2009: 3rd Workshop on Information Credibility on the Web in conjunction with 18th World Wide Web, Madrid, 20 April 2009, forthcoming.

Time will tell if this model will be used, if it would work in the real world, etc. But for sure there is some stir about the scholarly publishing:

• A blog (discussing similar themes, and one more), and another post on another blog (Update: another one).
• Three interesting pieces.
  
• Some interesting web pages.

S.