At my hospital, I’ve occasionally participated as a reviewer in a mock study section, aimed at teaching fellows and junior faculty about the process of grant review. It’s an enormously useful exercise for many junior colleagues, for whom the idea of a Great Study Section on High reviewing their work is both intimidating and mysterious. There are lots of valuable lessons to be learned as a fly on the wall while experienced reviewers discuss your work; I frankly wish we had the bandwidth to do more of these, particularly for folks who aren’t just starting out and are presumed to know what they’re doing already.
But these exercises often don’t include some critical aspects of the review process that arise when you take a mixed bag of supposed experts who fly in for the day (and buy their own lunches!) in order to participate in a crucial but underappreciated part of the grant process. That is, there are emergent properties of study sections that are not polite to discuss, but that can play a major role in whether or not a study is funded or buried deeper than deep in the ground. Those of us in the trenches might argue about how much these properties influence the process, but it’s almost impossible to deny that they exist altogether. A recent commentary in Nature encourages a shift to anonymous (blinded) grant applications, and motivated me to think about whether such a shift would really solve the problems in grant review.
I’m going to aim for a series of posts describing these worrisome features of the process. Before jumping in, a few admissions and caveats. As will be apparent, I believe our review process is fundamentally broken. I’ve come to this belief as a scientist who has been submitting grants to a variety of NIH institutes and study sections over two decades, and who has been very fortunate to remain continuously funded during this time. More to the point, I’ve sat on multiple study sections and chaired a few as well. And I’m moved to write because I believe these problems are only getting worse, and I believe that unless people who benefit from this system speak out, there’s little hope for improvement.
The caveats: anyone seeking to carry on an informed conversation about study sections is almost by definition limited to anecdote. The “Open Mike” blog at NIH (https://nexus.od.nih.gov/all/category/blog/) and others have done an admirable job aiming to answer questions related to grant review with data. But, as an informatician, I’d far prefer to lay my hands on the data myself, and ask and answer my own questions. Because of the confidentiality of the review process, such data are difficult to come by for non-NIH insiders. (And, based on offline conversations with such insiders, they’re quite limited in the extent to which they can address controversial topics with such data). So, caveat emptor: I can’t back my arguments up with numbers yet, though I’m working on it. Another limitation to consider is that, while I’ve had broader experience than most researchers I know in terms of institutes and types of study section, there’s still massive ascertainment bias at work. So, I have no doubt that others’ experience may vary substantially from my own.
One final caveat. While I believe strongly that NIH has an obligation to address these problems, I think changes need to come from the top down. The program officers and SRO’s at NIH are the heroes of the process, trying to make a fundamentally broken system a little less so. I’d liken them to the hosts of a dinner party: they can manage the invitations and decide who sits where, but it’s not their fault if the guests insist on breaking things or eating with their hands. It’s distressing to hear program staff sigh and say, “the reviews didn’t go the way we expected.” In particular, when study sections go off the rails in ways subtle and less so, staff are very limited in their ability to step in, except in the most egregious of cases. Indeed, given their responsibility to ensure a balanced playing field, like a hands-off NFL referee, the tendency is to refrain from calling penalties and ‘let ’em play the game’. But just as the NFL has been dragged kicking and screaming to implement rules to protect the players, perhaps NIH can begin to empower SRO’s to do the same.
So… join me on a whirlwind tour of the Four Reviewers of the Apocalypse:
brain conspiracy 
You must be logged in to post a comment.