Why is it bad to judge a paper by citation count?

2018-03-20 17:32:06

I hear a lot from the experts that citation count is a bad idea as a measure of judging a paper. This seems simply counter-intuitive to me. I would like to know if any study has been done in this direction?

I do not know of any study in this field, but I have a hunch that the following points apply (without any particular order, just numbered for convenience in the case of comments/replies):

Opportunity for citation: A paper that presents the ultimate answer to a problem is probably more valuable than a paper that presents just an unfinished attempt at solving the same problem. Yet, the latter is much easier to build upon (because it still leaves obvious extension points) and thus might be cited much more frequently.

Meaning of citations: There is no rule that a paper needs to be cited positively. If a paper has an obvious flaw, this may give rise to a number of other papers that just cite the flawed paper to point out what they are about to prove wrong or do bett

  • I do not know of any study in this field, but I have a hunch that the following points apply (without any particular order, just numbered for convenience in the case of comments/replies):

    Opportunity for citation: A paper that presents the ultimate answer to a problem is probably more valuable than a paper that presents just an unfinished attempt at solving the same problem. Yet, the latter is much easier to build upon (because it still leaves obvious extension points) and thus might be cited much more frequently.

    Meaning of citations: There is no rule that a paper needs to be cited positively. If a paper has an obvious flaw, this may give rise to a number of other papers that just cite the flawed paper to point out what they are about to prove wrong or do better.

    Citation scope: Referencing a paper does not mean referencing the whole paper, or its core finding. A reference to a paper might mean that the whole paper is pointed out as related work, or it could just mean that the sec

    2018-03-20 17:35:58
  • Citation count is a good example of a phenomenon subject to the Matthew Effect: a feedback process in which privileged individuals become more privileged as a result of their privilege. Quality, of course, does have a significant correlation with a paper drawing citations. Its citations are also strongly affected, however, by the fame of its authors, the fame of its venue, and also simply by the fact that other people cite the paper (thus making people like yourself judge it as "higher quality" and therefore more likely to cite it). Likewise, the mere fact that a paper is obscure does not mean that it is bad. Mostly, it simply means that nobody is paying attention to it. Finally, the fact that people are happy with a result doesn't necessarily make it true, as any number of scientific shifts in thinking (not to mention scandals) can attest; a paper may even draw citations precisely because other people are criticizing it.

    Citation count is still a good way of getting a good firs

    2018-03-20 17:42:26
  • tl;dr.

    It only works under certain conditions.

    There is actually a whole field of study that explores the scientific process via publication metrics, it's called bibliometrics. It is true that citation count within a field is sometimes used as a proxy to estimate an article's quality. See for example this recently published study about peer review:

    Siler K, Lee K, Bero L (2015) Measuring the effectiveness of scientific gatekeeping PNAS Jan 13;112(2):360-5. doi: 10.1073/pnas.1418218112

    From which the highlighted paragraphs below are copied.

    Arguments in favor

    First, the reasons given to consider citation count as a measure of quality (I edited the references to match the ones given at the bottom of this post):

    Scientists cite work for a myriad of reasons (1, 2). However, the

    vast majority of citations are either positive or neutral in nature

    (3). We worked with the assumption that scientists prefer to build

    upon other quality research with their own work. As Latour

    2018-03-20 17:46:54
  • First off, don't trust generalizations too much. The number of citations is not a perfect measure by far. First, more citations take time to accumulate, once could possibly look at citations per year or something similar. Second, it is possible to cite your own work so without filtering out so-called self-citations you may see inflated values. Self-citation, in itself is not necessarily an evil either, there are many reasons why one must reference ones own earlier work. One obvious reason is that later work often builds on earlier work and part of that is usually earlier work by the same person. Third, the number of citations are field dependent and in bibliometrics methods exist to remove such bias. Hence an article in a hot topic with much research will receive more hits than an article, no matter how excellent, in a small field.

    So, when judging an article from its citations, it is useful to keep the problems in mind and not over-interpret. This is essentially not different from

    2018-03-20 17:56:18
  • The reason citation counts are not good for judging paper quality is that there are a great many reaons (15, 28, 26 - depending on the study) for citing, and citation practice vary between disciplines. See:

    Case, D O & Higgins, G M 2000 How Can We Investigate Citation Behavior? A Study of Reasons for Citing Literature in Communication. JASIS 51(7) 635-645

    for references to earlier studies, and report of a study in the discipline of communication.

    2018-03-20 18:07:37
  • Others have mentioned, but it bears repeating - citations are not necessarily positive. Plenty of papers use faulty methodologies and are cited as examples of what not to do.

    Also, at least within the social sciences, citation is correlated with age and 'first mover advantage'. Because any academic paper worth its salt is going to cite previous work on the topic in the literature review, older papers will naturally garner higher citation counts, all else held equal. This has very little to do with the quality of those papers.

    2018-03-20 18:18:56
  • First of all, what does it mean to better?

    I know this may sound like nitpicking, but I can provide an example that gives a strong negative correlation between citations and a certain measure of better. I typically write algorithms for my research. Generally speaking, you don't get to publish unless your algorithm is an improvement in some meaningful way over previous algorithms. But you certainly need to cite previously published algorithms! Thus, for a given problem, the early algorithms will have a very large number of citations, while the later algorithms will have a much smaller number of citations.

    So if your definition of "better" is that the algorithm is faster (a very reasonable definition), then the papers with large numbers of citations are likely to be the worst in the literature (since they are the oldest).

    As an anecdote, I just published a paper on an algorithm that I have reason to believe will close the book on optimization for that particular problem (partly b

    2018-03-20 18:23:15
  • There have been many good answers to this question, but I wanted to frame what I see as the most important points.

    Citation counts measure impact not quality:

    The citation count of an article is a measure of research impact, and not quality. Even as a measure of research impact, it has issues, but in general, "research impact" is closer to what it represents. A high quality new paper will have minimal citations, because it has had minimal time to have an impact. Similarly, the quality of a paper is only loosely related to research impact. Some high quality papers are difficult for the literature to digest. Some low quality papers evolve into standard publications that are commonly cited in a given context. Entire literatures may use questionable methods but build a whole ecosystem of mutually reinforcing citations.

    Citation count is confounded by time since publication: One of the biggest points is that:

    citation_count = average_citations_per_year x years_since_publication

    Time

    2018-03-20 18:33:35
  • I think it boils down to the question:

    Do you use citations as a measure or as a metric.

    A measure is a value you get by measuring a quantity (e.g. miles, seconds, citations, words). A measure is a mostly objective attribute. A measure does not have any meaning, it is only a quantisation.

    A metric uses usually multiple measurements and combines them. After the combination the metric can be used and compared to a set of rules or other metered objects in order to sort or value them. The combination of measurements is the step were a) meaning and b) subjectivity enters the whole process.

    It is therefore absolutely understandable to use citations as a measure but you will most likely never use it as a metric itself. When the number of citations is used as a metric for paper-quality, assumptions need to be made which cannot be proven (except by measuring something else as well).

    Only because one can measure something does not mean that the measurement contains any information.

    2018-03-20 18:46:24
  • There was a paper published in 1994 in a medicine journal entitled A mathematical model for the determination of total area under glucose tolerance and other metabolic curves which describes a method for computing the area under a curve, dubbed "Tai's method". This paper got 311 citations according to Google Scholar, which isn't too bad for medicine papers apparently (although some can rake in literally 1,000 citations...).

    This method is the trapezoidal rule and according to Wikipedia, it was known to ancient Babylonians in 50 BC. Every undergrad with a vague math education, perhaps advanced high school students, or someone who reads a pop math blogs, knows about it.

    You wouldn't know this insanity from just looking at the citations of the paper. There are many good arguments in the other answers, but if this isn't damning enough to make you look at citation counts with some circumspection, I don't know what will.

    2018-03-20 18:56:10
  • Well, a couple of reasons: The Tree of Knowledge within Academia is far from perfect and is rooted in several ontological biasiis about the world. A wrong premise can result in systemic error in the whole works. Following them perpetuates a possible negative feedback loop.

    That being said, Google has been quite successful in creating a useful tool to sort out the chaff by using the equivalent of citation counts to order your search results.

    What does it miss? The lone, iconoclast who might be a genius. Einstein, for example, probably would have been missed, using citation only.

    2018-03-20 19:05:19
  • Human beings, regardless of intelligence level, generally tend to subscribe to the "herd mentality". I guess it's because we, as animals like any other animal, want to conserve energy. For example, if everyone believes Stephen Hawking or Lisa Randall is rarely wrong, then it's in my best interest to focus on what he/she has to say. It will save me the energy of exploring other sources and finding answers for myself.

    At someone's request I will try to clarify my answer.

    Let's say I'm writing a paper about black holes. Obviously, I want to write the best possible paper but I'm not the best physicist on planet Earth. The best physicists on planet Earth will probably have the most citations on the subject matter. Does having the most citations about a subject mean that the highly cited individual is always correct concerning the subject? No. I have to realize that if I'm referencing that person's work. Assuming that Stephen Hawking is right all the time about black holes or even assumi

    2018-03-20 19:12:19