Alcune bufale sull'università italiana

Alberto Baccini su ROARS elenca alcune affermazioni sull'università italiana, affermazioni che si sentono spesso e che sono, come dice appunto Baccini, false e/o distorte -- insomma, "bufale":

Il contesto. Ecco ciò-che-tutti-sanno-dell’università-italiana (poco importa che alcuni punti dello scenario siano sostanzialmente falsi e altri ingigantiti e distorti):
1.  “La ricerca scientifica attraversa un periodo di stasi”, perché l’università produce poca ricerca [si legga qui e qui], ed è avviata al declino;
2. “La ricerca scientifica deve servire alla scienza e alle esigenze nazionali. Non deve servire a creare nuove cattedre e nuovi insegnamenti.” Il declino della ricerca italiana è causato dall’autoreferenzialità dei baroni.
3. Il sistema di reclutamento è distorto e corrotto da nepotismo e clientele. Il merito è mortificato.
4. All’università italiana e alla ricerca non mancano le risorse. La ricerca condotta dai baroni è spesso inutile per la società ed autoreferenziale.
5. I baroni hanno stipendi tra i più alti al mondo.
Da questo segue che l’università italiana è irriformabile con gli strumenti legislativi usuali; c’è bisogno di una rivoluzione (lo sostiene per esempio  Andrea Ichino). La politica ha il compito di individuare una élite accademica illuminata e d’avanguardia che possa modificare dall’alto il funzionamento della università e della ricerca italiana. I due snodi fondamentali sono finanziamento e reclutamento. Lo strumento istituzionale è l’ANVUR: un organismo tecnico di nomina ministeriale, lasciato incompiuto dal governo di centro-sinistra, cui vengono attribuiti poteri (oltre a molti altri) su valutazione e criteri per il reclutamento.

S.

È tutto un magna magna

Citation cartel.

S.

Post, P.S. e commenti

A volte il P.S.

Ps. A seguito del mio post su Valutazione e burocrazia mi è stato segnalato che, secondo sviluppi recenti di cui non ero a conoscenza, è stato deciso che nella prossima tornata del Rae non si farà uso di un sistema automatico di valutazione.

e i commenti sono più importanti dei post.

S.

make.tenure.FAST

:-)

S.

Research evaluation and bibliometrics - again

Another REF report. Quotations:

bibliometric indicators alone cannot provide a sufficiently robust measure of quality to drive funding allocations in any discipline at present. (p. 27, emphasis added)

Through those initial consultations – and the more recent bibliometrics pilot exercise – a widespread consensus has emerged that while metrics should inform expert review they are not sufficiently robust to replace expert review. While we remain concerned to reduce the burden of assessment, we believe we have exhausted the main options for any radically different alternative approach. The REF will be driven by a process of expert review, informed by metrics. (p. 27, emphasis added)

As I wrote time ago: it is stupid not to rely on bibliometric indicators. And it is stupid to rely on bibliometric indicators only.

S.

On peer review and scholarly publishing - again

Another interesting proposal, although my own papers are not cited ;-)

S.

Invited talks

I'll be giving two invited talks in the next months:

• Readersourcing: Scholarly publishing, peer review, and barefoot cobbler's children @ FDIA 2009 @ ESSIR 2009, 01/09/2009.
  
  Abstract:
  
  I will start from an introduction to the field of scholarly publishing, the main knowledge dissemination mechanism adopted by science, and I will pay particular attention to one of its most important aspects, peer review. I will present scholarly publishing and peer review aims and motivations, and discuss some of their limits: Nobel Prize winners experiencing rejected papers, fraudulent behavior, sometimes long publishing time, etc. I will then briefly mention Science 2.0, namely the use of Web 2.0 tools to do science in a hopefully more effective way.
  
  I will then move to the main aspect of the talk. My thesis is composed of three parts.
  
  (i) Peer review is a scarce resource, i.e., there are not enough good referees today. I will try to support this statement by something more solid than the usual anecdotal experience of being reject because of bad review(er)s --- that I'm sure almost any researcher has experienced.
  
  (ii) An alternative mechanism to peer review is available right out there, it is already widely used in the Web 2.0, it is quite a hot topic, and it probably is much studied and discussed by researchers: crowdsourcing. According to Web 2.0 enthusiasts, crowdsourcing allows to outsource to a large crowd tasks that are usually performed by a small group of experts. I think that peer review might be replaced --- or complemented --- by what we can name Readersourcing: a large crowd of readers that judge the papers that they read. Since most scholarly papers have many more readers than reviewers, this would allow to harness a large evaluation workforce. Today, readers's opinions usually are discussed very informally, have an impact on bibliographic citations and bibliometric indexes, or stay inside their own mind. In my opinion, it is quite curious that such an important resource, which is free, already available, used and studied by the research community in the Web 2.0 field, is not used at all in nowadays scholarly publishing, where the very same researchers publish their results.
  
  (iii) Of course, to get a wisdom of the crowd, some readers have to be more equal than others: expert readers should be more influential than naive readers. There are probably several possible choices to this aim; I suggest to use a mechanism that I proposed some years ago, and that allows to evaluate papers, authors, and readers in an objective way. I will close the talk by showing some preliminary experimental results that support this readersourcing proposal.
  
  Disclaimer: This talk might harm your career; don't blame me for that.
  
  P.S. Yes, this is somehow related to a previous post...
  
  
• Two Tales on Relevance Crowdsourcing: Criteria and Assessment @ GIScience Colloquium, University Zurich-Irchel, 13/10/2009.
  
  Abstract (DRAFT):
  
  In Information Retrieval (IR) and Web search, relevance is a central notion. I will discuss how to outsource to the crowd two relevance-related tasks. The first task concerns the elicitation of relevance criteria. After some results obtained in the 90es, relevance criteria (i.e., the features of the retrieved items that determine their relevance) seem well known and stable. We conjectured that for e-Commerce / product search the criteria might be different, and we used Amazon Mechanical Turk, a crowdsourcing platform, to find a confirmation of our hypothesis.
  
  The second task concerns effectiveness evaluation. A common evaluation methodology for search engines and IR systems is to rely on a benchmark (a.k.a. test collection); benchmarks need relevance assessment, i.e., to assess the relevance of documents to information needs; usually this task is done by experts, either paid for their work or participating in the evaluation exercise themselves. Again, we used Mechanical Turk, this time to re-assess some TREC topic/document pairs and thus see if we can "get rid of" relevance assessors by replacing them with a crowd working remotely on the Web. I'll discuss the preliminary results on the reliability of the crowd of assessors.
  
  (this is joint work with Omar Alonso, A9.com; thanks also to Dan Rose)
  

I'll publish the slides here, once ready. Meaning: after the talks :-)
S.

Research evaluation & bibliometrics

At my university there is a strong push, by... someone, to use bibliometric indicators to evaluate research productivity. My position is:

• Bibliometrics is useful. It would be crazy not to use it.
• Bibliometrics is not enough. It would be equally crazy to rely on bibliometrics only. A more general, scientometrics-based approach, should be used.
  
• Expert peer review, informed by bibliometrics but not only, is the only reliable evaluation mechanism.
• Evaluation needs resources (money, people, time, data, ...).
• Evaluation strategies and approaches should be decided by evaluation experts, not by beginners.
  

Probably, someone, somewhere in the world, is fighting trying to convince they colleagues and/or administrators, that bibliometrics is useful. At my university, I'm in the opposite position: I'm fighting trying to convince people that bibliometrics can give some indication, but it's just one of the parameters to be measured, and that it is stupid to rely on bibliometrics only.

BTW, so far I did not get much success - they just pretend I'm not saying anything. Well, actually they've changed from using bibliometrics to evaluate single researchers (a position that demonstrated their level of... erm let's be polite and say "expertise"...) to using bibliometrics to evaluate groups of researchers. But I'm stubborn ;-)

Anyway, I'm writing this post because I'm happy that in the last REF report one can read:

There was a strong consensus that bibliometrics are not sufficiently mature to be used formulaically or to replace expert review, but there is considerable scope for citation indicators to inform expert review in the REF.

Let's see if bibliometrics enthusiast at my university will take this into account, or just pretend it doesn't exist.

And, yes, of course, let's see if bibliometrics-retractors will take this into account, or just pretend it doesn't exist.

S.