I’ve previously written about how the current administrative structures that support scientific progress in the US evolved, and why the current moment presents a real opportunity for positive evolution. But what does that change look like? How do we re-imagine the system for allocating funds in a way that simultaneously supports innovation and strengthens our ability to manage scarce resources? The only real way to answer these questions is to apply the tools of science, just as we would to solve any empirical problem. What has emerged from that effort is growing evidence that we can use large-scale scientific behavior as a lens for determining which topics are, or are not, worth pursuing. To see how that would work in practice, let’s start with a quick overview of the prevailing model for choosing which ideas receive support in the form of funding.

The journey of a scientific advance currently begins with an individual scientist requesting support for a particular idea. That request, in the form of a grant application, is sent to a panel of experts who give their opinions on its relative merits. Although review panels are commonly composed of two dozen or more scientists, in practice just two or three individuals typically determine the fate of any given proposal. This system was developed when science was much smaller, but both habit and necessity have conserved its form to a remarkable extent over many decades. The wisdom in soliciting expert opinion before making a consequential decision is clear, and convening a panel has historically been the way to do it. However, different experts may have different, equally valid perspectives on a proposal that are rooted in their individual experiences and expertise. In a resource-constrained environment those differences can generate enough noise to overwhelm any signal that flags a proposed line of inquiry as likely to produce important results [1-4]. We might attempt to improve the signal-to-noise ratio by asking for more opinions, but this is logistically unrealistic due to both practical constraints on the size of review meetings and the need to arrive at a verdict in a timely fashion.

In their classic work on the sociology of science, Latour and Woolgar establish that scientists are motivated to take up or abandon a research problem based in part on their belief that its solution will represent a major advance [5]. The movement of practicing scientists towards or away from a specific area of research, then, provides an alternative means of assessing expert opinion that can be implemented at scale. The system for organizing scholarly communications described in Davis et al. [6] is easily adapted to this purpose. The borders of a research topic are defined by the networks generated by subject matter experts when pairs of papers are cited together; growth and stagnation are simply measured by the percentage of new papers that fall within those borders. The end result is that instead of being limited to asking what a handful of scientists think, we can observe what thousands actually do. Although still not infallible, this information can meaningfully guide and inform the judgment of decision-makers.

One obvious way to operationalize this concept would be for funders to concentrate their grantmaking on topics that are predicted to produce future breakthroughs [6]. However, breakthroughs are rare, and for the scientific enterprise to remain healthy, research investment must span a broad range of topics. Since our approach to predicting breakthroughs begins by systematically and agnostically identifying all topics, it is straightforward to collect the subset that correspond to any particular problem – something as broad as “cancer” or as specific as hemochromatosis – and quantify the novelty (measured as age since first appearance), relative growth, and existing level of organizational investment for each (Fig. 1).

Figure 1. Cancer-relevant research topics, isolated from the co-citation network of all papers in PubMed by regularized Markov clustering. Each bubble represents a unique scientific topic (n = 687). Rate of progress integrates information on the number and field-normalized influence of papers on a given topic, both features of the breakthrough signal described by Davis et al. [6]. Age indicates the earliest detectable appearance of the topic depicted. One novel and rapidly advancing topic, the role of lipid metabolism, inflammation, and a high fat diet in cancer, is highlighted. MS in prep.

Using data to identify specific topic(s) that have only recently appeared in the literature provides a way to ensure that funders invest in genuinely new ideas rather than those its staff are merely encountering for the first time. It also acts as a bulwark against a common failure mode that plagues programmatic initiatives, specifically, the development of a targeted funding opportunity that fails to attract any applications simply because no established constituency for it exists [7]. Finally, since the likelihood of a submission increases exponentially as a funding opportunity becomes more similar to an applicant’s area of expertise [8], knowing the size of the likely applicant pool allows funders to control demand more broadly, providing a potential mechanism to reduce hypercompetition (Fig. 2).

Figure 2. Probability of application as a function of the similarity between a given funding opportunity announcement (FOA) and the applicant’s prior publications. Applications from 110,000 scientists to 390 RFAs or PAS that made use of the R01 mechanism and included set-aside funds (solicited R01s) were analyzed; similarity was evaluated based on the use of terms from the Medical Subject Headings (MeSH) thesaurus. Applications made under the American Recovery and Reinvestment Act (ARRA) of 2009 were excluded. Figure reproduced from 20 Myers [8].

It is immediately apparent that data like those shown in Figure 1 also provide a way for funders to identify topics that may already be adequately represented in a portfolio. Since funding decisions are inherently value judgments, they should never be based on computational analysis alone; decisions to reduce commitments in a specific area should only be made after further review by human authorities who can be held accountable for their choices. An area of research that appears to be making slow progress may be conducting clinical trials, contributing valuable technological development through the patent process, maintaining useful community resources, or some combination thereof. Divestment from an area should therefore be approached with caution, simply because innovative new areas of research may be born when a body of knowledge from larger, older fields is re-interpreted in light of one or more new discoveries.

Within the bounds of these caveats, we need to acknowledge that the study of some topics continues to attract funding well after the work has reached a point of diminishing returns. Discarding old ideas is an integral part of scientific progress; fields naturally lose momentum as new evidence and models supersede the old, and a reduction in the number and influence of papers on a given topic is one sign of decay [9]. We cannot expect to support new areas through budget increases alone. Indexing the reallocation of resources to the scholarly activity of the scientific community, using a carefully field-normalized metric like the Relative Citation Ratio [10], can help ensure that scarce dollars are not disproportionately spent on incremental advances. It should go without saying that any topics flagged by data, whether for deprioritization or support, should be fully disclosed to applicants up front, before they commit their time and effort to preparing and submitting applications.

A concern that commonly arises in conversations about incorporating data into decision-making is that people and resources might be funneled into areas that currently show promise at the expense of those whose potential will become apparent only at a later date; that is, we might pick the “wrong” areas at the expense of the “right” ones. This argument overlooks the fact that our current system already distributes resources unevenly across the scientific landscape based on an imperfect ability to predict the future [11]. It also assumes that the scientific ecosystem is perfectly elastic, which it is not. Changing fields is not frictionless; the entry of researchers into a new area is constrained by their training and expertise. Additionally, since the scientific focus of many researchers and donors reflects a personal connection to a given problem, they will maintain a constant focus on it regardless of how distant a solution might seem.

The true risk, as Vannevar Bush wrote, comes from the daft belief that if one does nothing one does not make mistakes [12]. Our fear that any change we make may be for the worse has left us unable to substantially reform decision-making processes that we know are flawed. Data isn’t a perfect panacea for all the problems of the research enterprise, but including it in our funding deliberations does not mean dismantling peer review or surrendering consequential choices to an algorithm. It simply means providing people already involved in decisions with an orthogonal way of checking some of their assumptions. Would that lead to better outcomes? The only way to find out is to try it and see.

Share

References

[1] D. Kaplan, N. Lacetera, and C. Kaplan, “Sample Size and Precision in NIH Peer Review,” PLOS ONE, vol. 3, no. 7, p. e2761, Jul. 2008, doi: 10.1371/journal.pone.0002761.

[2] E. L. Pier et al., “Low agreement among reviewers evaluating the same NIH grant applications,” Proc. Natl. Acad. Sci., vol. 115, no. 12, pp. 2952–2957, Mar. 2018, doi: 10.1073/pnas.1714379115.

[3] P. S. Forscher, M. Brauer, F. Azevedo, W. T. L. Cox, and P. G. Devine, “How many reviewers are required to obtain reliable evaluations of NIH R01 grant proposals?.,” 2019. doi: https://doi.org/10.31234/osf.io/483zj.

[4] V. E. Johnson, “Statistical analysis of the National Institutes of Health peer review system,” Proc. Natl. Acad. Sci., vol. 105, no. 32, pp. 11076–11080, Aug. 2008, doi: 10.1073/pnas.0804538105.

[5] B. Latour and S. Woolgar, Laboratory Life: The Construction of Scientific Facts. Princeton University Press, 1986.

[6] M. T. Davis et al., “Prediction of transformative breakthroughs in biomedical research,” 2025. doi: https://doi.org/10.64898/2025.12.16.694385.

[7] J. Lorsch, “NIH’s Path to a Simpler Funding Opportunity Landscape.” [Online]. Available: https://web.archive.org/web/20260330013627/https://grants.nih.gov/news-events/nih-extramural-nexus-news/2026/03/nihs-path-to-a-simpler-funding-opportunity-landscape

[8] K. Myers, “The Elasticity of Science,” Am. Econ. J. Appl. Econ., vol. 12, no. 4, pp. 103–34, 2020, doi: 10.1257/app.20180518.

[9] C. K. Singh, E. Barme, R. Ward, L. Tupikina, and M. Santolini, “Quantifying the rise and fall of scientific fields,” PLOS ONE, vol. 17, no. 6, p. e0270131, Jun. 2022, doi: 10.1371/journal.pone.0270131.

[10] B. I. Hutchins, X. Yuan, J. M. Anderson, and G. M. Santangelo, “Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level,” PLOS Biol., vol. 14, no. 9, p. e1002541, Sep. 2016, doi: 10.1371/journal.pbio.1002541.

[11] T. A. Hoppe et al., “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists,”Sci. Adv., vol. 5, no. 10, p. eaaw7238, doi: 10.1126/sciadv.aaw7238.

[12] V. Bush, Pieces of the Action. Stripe Press, 2022.

Any discussion of what it takes to produce a scientific breakthrough requires a few words about the current system for supporting scientific discovery, specifically, how that system developed and why it’s no longer performing as intended. Some very thoughtful people have made good suggestions about how to fix it, but I’ll finish by sharing mine, which is a little bit different.

Let me start with a brief summary of how the current system evolved. Federal dollars are the largest source of research and development funding for academic institutions in the United States, and have been since 1953 [1]. There are many agencies that distribute and administer federal research grants, but the two largest, the National Institutes of Health (NIH) and the National Science Foundation (NSF), share a common ancestor, the Office of Scientific Research and Development (OSRD). OSRD was established in 1941 by executive order and Vannevar Bush, at that time the President of the Carnegie Institution of Washington, was named as Director, reporting directly to President Franklin D. Roosevelt.

In November 1944, Roosevelt wrote a letter to Bush, asking what could be done to maintain the wartime pace of scientific advancements after peace was secured. In response, Bush wrote the now-famous report, Science, the Endless Frontier, persuasively arguing that advances in science lead to new industries, more jobs, and a higher quality of life. Any money the government spent advancing science would be worth it, he argued, because the return on investment would be so high – and subsequent analysis has consistently shown this to be true [2].

To help make his case, Bush also pointed to two discoveries – penicillin and radar – that helped save Allied lives in the war against fascism. No one set out to discover either; both were byproducts of scientists following their curiosity about the world. Lucky accidents. Serendipity. We couldn’t rely on luck, Bush said. We needed to invest, and because we couldn’t know where the next penicillin would come from, we needed to invest broadly.

Thoroughly convinced by The Endless Frontier, Congress voted to establish the NSF in 1950, largely following Bush’s design. A notable exception was the rescission of biomedical research, which Bush had intended to be within the remit of NSF but which Congress instead assigned to the nascent National Institutes of Health. By any measure, the enterprise that Bush dreamed of has grown to be wildly successful. To date, NIH has supported 174 Nobel laureates, while NSF has supported 268 [3,4]. The benefits haven’t only been academic. NIH funding has been essential in the development of nearly every new drug that goes to market [5], while NSF research undergirds the lasers in LASIK eye surgery, the touchscreens and batteries in smartphones, the computer-aided design that 3D printing relies on, and so much more.

But there was a flaw in Vannevar’s plan – a problem he didn’t foresee that has grown over time. The scientific frontier is endless; our budget is not. It’s easy to say Congress should provide more funds, but just like widening highways actually worsens congestion, pumping in more money expands the frontier and recreates the original problem. Nor does it address the downstream effects of a hyper-competitive research environment, many of which were succinctly described by Mike Lauer, the former Director of Extramural Research at NIH, in a recent interview with Santi Ruiz of Statecraft. As Dr. Lauer acknowledges, these include compromised rigor, reduced innovation, and a raft of perverse incentives related to training young scientists. Two things can be true at once: the current system has enabled great achievements, and we’re leaving advances on the table. We need to do better.

Midway through the interview, Ruiz suggests experimenting with the way we fund science, and Lauer agrees. Their discussion focuses on a single proposal: shifting away from smaller project grants to individual investigators in favor of large block grants. I have an alternative proposal: use data [6], rather than relying on expert opinion alone, to identify potential breakthroughs and other emerging areas that would benefit from early investment.

Figure. The development of research on fluorescent proteins and super-resolution fluorescence microscopy in the form of a trajectory, from 1995 (the first year the topic existed as a discrete cluster of publications) through 2017. The area of each cluster (circle) is proportional to the number of publications it contains, benchmarked to the size of the largest (2016) cluster of 4,279 publications. The blue asterisk indicates the cluster in which the breakthrough papers, which were published in 2006, first appear. From 2025 Davis et al. [6].

Arguably, this is where the existing system is most broken: it’s well established that unfamiliar and anti-paradigmatic ideas are viewed with skepticism. Studies have found that as grant proposals become more novel, reviewers uniformly and systematically assign them worse scores [7]; recognition of novel research papers is delayed, and is more likely to come from investigators working outside their field of origin [8]. Most seriously, seeing their peers provide negative evaluations drives reviewers to be more critical themselves, so that the actual process of peer review is the source of its conservatism [9].

To truly address our current problems and accelerate scientific progress, we need to do more than change who gets funded; we need to change how funding decisions are made.

Share

References

[1] https://web.archive.org/web/20260110144801/https://ncses.nsf.gov/

[2] T. Gullo, B. Page, D. Weiner, and H. L. Williams. Estimating the economic and budgetary effects of research investments. NBER working paper 33402 DOI 10.3386/w33402 (2025)

[3] https://web.archive.org/web/20260111024113/https://www.nih.gov/about-nih/nih-almanac/nobel-laureates

[4] https://web.archive.org/web/20260111024929/https://www.nsf.gov/about/history/nobel-prizes

[5] E. G. Cleary, J. M. Beierlein, N. S. Khanuja, L. M. McNamee. and F. D. Ledley. Contribution of NIH funding to new drug approvals 2010-2016. PNAS 115: 2329-2334. (2018)

[6] M. T. Davis, B. L. Busse, S. Arabi, P. Meyer, T. A. Hoppe, R. A. Meseroll, B. I. Hutchins, K. A. Willis, and G. M. Santangelo. Prediction of transformative breakthroughs in biomedical research. bioRxiv, doi:https://doi.org/10.64898/2025.12.16.694385 (2025)

[7] K. J. Boudreau, E. C. Guinan, K. R. Lakhani, and C. Riedel. The Novelty Paradox & Bias for Normal Science: Evidence from Randomized Medical Grant Proposal Evaluations. HBS Scholarly Articles, http://nrs.harvard.edu/urn-3:HUL.InstRepos:10001229 (2012)

[8] J. Wang, R. Veugelers, and P. Stephan. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy 46:1416-1436. (2017)

[9] J. N. Lane, M. Teplitskiy, G. Gray, H. Ranu, M. Menietti, E. C. Guinan, and K. R. Lakhani. Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation. Management Science, 68:4478-4495. (2022)

As I noted in my brief inaugural post here at The Progress Bar, there are lots of reasons I’m excited about 2026. A big one is that I anticipate having more time to read, think, and write about what it takes to produce a scientific breakthrough. It’s somewhat surprising to me that so many practicing scientists, who are devoted to using data to unravel the mysteries of life and the universe, seem to have consigned this question to the realm of the unmeasurable. That might once have been true, but recent advances in data science and computational power have changed the game.

If you’re here, it’s safe to say you too are curious about measuring and predicting scientific progress. Having the opportunity to share my thoughts with you is another reason I’m looking forward to the coming year. Before we get started in earnest, though, it seems like a good idea to tell you a little about my philosophy, my approach to this problem, and why I think now is the time to tackle it.

First, the philosophy. Science is the practice of making reproducible observations about a defined system, with the intent to use those observations to formulate testable predictions. That first part – making reproducible observations – is about describing your system. This is an important and necessary prerequisite to predicting its behavior, but if a field gets stuck here, it is not doing effective science. Neither is a field doing effective science if its observations don’t reproduce, or if it requires practitioners to continually violate Occam’s razor by cutting their system into smaller and smaller pieces to preserve the dominant paradigm.

Assuming you’re describing a system for the first time (or you’ve chucked all your previous attempts and are starting fresh), there are arguably two ways to go about it – reductionism, or holism. The reductionist approach has a lot of attractions: isolate one factor, control the conditions, characterize it down to the atom. When done well, this can provide truly useful information, as in the case of structural biology. Of course, there are also limitations: the temptation to concentrate on facets of the problem that are easy to measure but functionally unimportant can be strong. Worse, like blindfolded scientists studying an elephant, focusing on just one part of the whole can lead to some dramatic mischaracterizations.

The alternative is to take a holistic approach: rather than concentrating on a leg, consider the entire animal. For better or worse, this is my preference, because it beautifully accommodates the starting premise that as humans, we are all somewhat stupid. We may not know the full list of parts that make up the whole we’re studying, and we certainly don’t know which are the important ones. If we’re clever about the questions we ask, nature will tell us what matters, although it may take us a while to understand the answer.

To fully realize their promise, then, this is my definition of what studies of scientific progress should be – rigorous, predictive, interrogating relationships rather than parts.

The major challenge for holistic approaches is that they require capturing, storing, and analyzing enough high-dimensional data to accurately represent reality. If this is the roadblock, our current opportunity is obvious. When Vannevar Bush wrote Science, the Endless Frontier in 1945, ENIAC filled a room. In 1964, when the Institute for Scientific Information published the first Science Citation Index, the world’s fastest supercomputer, the CDC6600, weighed close to six tons and had a top speed of three megaFLOPS. Today, a run-of-the-mill Dell desktop workstation is over 100,000 times faster, and at a compact 17” x 8.5” x 21”, only slightly larger than a carry-on roller bag.

Access to data sources has improved, too. Records of NIH and NSF awards have moved online. A majority of scholarly literature is now digital-first, with metadata that is structured and machine-readable; optical character recognition (OCR) does the heavy lifting of filling in much of the historical record. CrossRef, Unpaywall, and the Open Citation Collection have brought the primary currency of academic credit into the public domain [1]. Natural language processing can extract central semantic themes from a corpus of documents far too large for any one human to read in its entirety.

The idea to put these pieces together has been in the air for a while now. Writing for Science in 2017, Aaron Clauset, Daniel Larremore, and Roberta Sinatra identified a widespread belief that computational methods could predict incipient advances more objectively and accurately than expert opinion [2]. They also very shrewdly described many of the challenges associated with attempts at prediction, including the risk that poorly designed efforts could actually increase inequality, penalize novelty, and inhibit innovation. Four years later, James Weis and Joseph Jacobson shared an effort to predict impactful research in biotechnology, using a random forest classifier to sort through a subset of papers pre-selected by human subject matter experts for their likely impact [3]. Much of the discussion at the time, correctly in my opinion, centered on the potential for this approach to reinforce the Matthew Effect; grounding a model on legacy metrics like journal impact factor and h-index is convenient, but the new construct unavoidably inherits the known biases of the old components. It is true, though, that you have to start somewhere, and early efforts are often flawed.

Last month, my colleagues and I released a new paper predicting breakthroughs across all domains of biomedicine [4]. Our network-based approach relies exclusively on article-level assessment; we didn’t pre-judge which work might have value based on journal of publication or author history. Surveying past breakthroughs in fields as diverse as biophysics, genetics, cell biology, and clinical care, we found four common features that together act as the signal of an impending breakthrough, detectable more than five years, on average, in advance of the subsequent publication(s) that announce the discovery. Since science should make testable predictions, we screened papers published between 2014-2017 for our signal of likely future breakthroughs. Early returns look promising; one of our picks has already won a Lasker award.

I’m eager to see how our predictions continue to play out over the next few years. The work also opens lots of new questions I find interesting. For example, the time between the signal of an upcoming breakthrough and the breakthrough itself is variable; is that influenced by the number of researchers focused on the problem, by the availability of grant funding, or both? Or maybe neither? Our data also suggest that breakthroughs are more likely to occur when fields converge or diverge; does the density of ideas adjacent to an area of research influence its breakthrough potential? Answering these and other questions about how discoveries are made will help us create the conditions that promote them, and now is the time to do it.

Thank you to everyone who has already subscribed to this new venture; if this is your first visit, I hope you’ll sign up for more.

Subscribe now

References

[1] B.I. Hutchins, K. L. Baker, M. T. Davis, M. A. Diwersy, E. Haque, R. M. Harriman, T. A. Hoppe, S. A. Leicht, P. Meyer, and G. M. Santangelo. The NIH Open Citation Collection: A public access, broad coverage resource. PLoS Biology, doi: 10.1371/journal.pbio.3000385 (2019).

[2] A. Clauset, D. B. Larremore, and R. Sinatra. Data-driven predictions in the science of science. Science, 355:477-480 (2017).

[3] J. W. Weis and J. M. Jacobson, Learning on knowledge graph dynamics provides an early warning of impactful research. Nature Biotechnology, 39:1300-1307 (2021).

[4] M. T. Davis, B. L. Busse, S. Arabi, P. Meyer, T. A. Hoppe, R. A. Meseroll, B. I. Hutchins, K. A. Willis, and G. M. Santangelo. Prediction of transformative breakthroughs in biomedical research. bioRxiv, doi:https://doi.org/10.64898/2025.12.16.694385 (2025)

Stability would not be high on my list of descriptors for this year.

Despite the challenges, I find myself looking into 2026 with excitement and optimism. A lot has changed in 80 years. We have access to tools and data that Vannevar and his contemporaries (probably) couldn’t have dreamed of; yet we’ve remained stuck, in a lot of ways, in the framework they set up. Some fairly basic questions about how best to produce innovations that promote human health and flourishing remain unanswered; maybe the extent of our success made it seem unimportant to ask.

I’ll be asking those questions in the coming year, exploring how we can use data to understand scientific progress, and ideally, how we can leverage that understanding to produce more breakthroughs in less time. I hope you’ll join me as I explore here at The Progress Bar.

Subscribe now