Soft peer review: social software and distributed scientific evaluation

The contribution that social bookmarking systems can provide to the problem of usage-based metrics for scientific evaluation is analyzed and it is suggested that collaboratively aggregated metadata may help fill the gap between traditional citation-based criteria and raw usage factors.

Abstract

The debate on the prospects of peer-review in the Internet age and the
\nincreasing criticism leveled against the dominant role of impact factor
\nindicators are calling for new measurable criteria to assess scientific quality.
\nUsage-based metrics offer a new avenue to scientific quality assessment but
\nface the same risks as first generation search engines that used unreliable
\nmetrics (such as raw traffic data) to estimate content quality. In this article I
\nanalyze the contribution that social bookmarking systems can provide to the
\nproblem of usage-based metrics for scientific evaluation. I suggest that
\ncollaboratively aggregated metadata may help fill the gap between traditional
\ncitation-based criteria and raw usage factors. I submit that bottom-up,
\ndistributed evaluation models such as those afforded by social bookmarking
\nwill challenge more traditional quality assessment models in terms of coverage,
\nefficiency and scalability. Services aggregating user-related quality indicators
\nfor online scientific content will come to occupy a key function in the scholarly
\ncommunication system.