Too Much Science, Not Enough Disruption?
Has truly disruptive innovations slowed down over time even when scientific output has increased exponentially? This is my favorite topic to mull over but this recent paper in Nature really brings clarity to this question and is fascinating read: 🧵 https://www.nature.com/articles/s41586-022-05543-x
This topic has been discussed a lot by likes of @peterthiel, @EricRWeinstein (look up their talks). Some blame on “publish-or-perish”, some on academic degradation while a charitable view is gradual disappearance of “low hanging fruit” in science. Authors reject these theories.
In this paper, author uses measure called CD index invented in 2017. It ranges from -1 (truly consolidating) to 1 (truly disruptive). The idea is that a consolidating paper is more likely to be cited along with previous works while disruptive ends the citation of what it cites.
CD index has been thoroughly validated. For ex, DNA Helix paper has CD of 0.62 while electron structure paper has CD of -0.22. Both won Nobel and got similar citation count in 5 years but former trashed all other theories while later used existing theory to build new one.
Decline in disruptive science is very real and measurable: Average CD decreased from 0.30 to 0.06 for Computer Science in span of 30 years, 0.36 -> 0 for physical sciences over 65 years, 0.38 -> 0.03 for medicine over 30 yrs!
Curiously, number of highly disruptive works have remained constant over the years. However as number of papers pile up exponential CD approaches 0. Authors also reject that this is due to drop in quality because pattern reoccurs even when restricting to Nobel papers, Nature.
Instead, authors hypothesize based on data that the disruption decline is rather due to too narrowly focused domain which eliminates cross-domain diversity necessary for disruption, relying only on familiar knowledge and struggle to keep up with new works.
In other words, our culture encourages safely earning that little bump in the human knowledge rather than taking risks. So, we get exponentially more tiny little bumps. True risk takers and out of box thinkers have remained roughly constant in exponentially increased population.