18: Brian Nosek
Welcome to Circle of Willis! For this episode I'm sharing a conversation I had a while ago with BRIAN NOSEK, professor of Psychology here, with me, at the University of Virginia, as well as co-Founder and Executive Director of the CENTER FOR OPEN SCIENCE, also here in Charlottesville. Brian earned his PhD at Yale University way back in 2002, only about a year before I first met him here, when I was just a jittery job candidate. Brian has been in the public eye quite a lot in the past decade or so, not only due to his work with the Implicit Association Test, otherwise known as the IAT, but also and perhaps mainly for his more recent path breaking efforts to increase the transparency and reproducibility of the work scientists do. I think you'll find that in our conversation, Brian is relentlessly thoughtful about everything that comes up. And I want to say here, publicly, that I think he's absolutely right, at the very least, about the toxicity of the current system of incentives and rewards faced by academic scientists. Occasionally you'll hear that "science is broken." It's a great, click-baity phrase that thrives in our current social media ecosystem. But it's completely wrong. Science is not and has never been broken. Even now, science is our most precious, life affirming, life saving, human activity. Literally nothing humans have invented has done more than science has to improve our welfare, to increase our sensitivity to the natural world, or to reveal the forces and mechanisms that form and constrain our miraculous universe. But the institutional structures within which science is done are in bad shape. At the foundation, public funding for science is dismal, and that problem is yoked to the steadily declining public commitment to higher education in general. Our institutions have come to rely on bloated federal grants to just keep the lights on, and the responsibility for securing those federal dollars has fallen heavily on the shoulders of scientists who ought to be focused on making discoveries and solving the world's problems. And because that is a heavy burden, institutional structures have formed to incentivize -- some would say coerce -- scientists into striving for those federal dollars. Want to get tenure? Better bring in some big federal grants. Want 12 months of continuous salary? Better bring in some big federal grants. You get the idea. But there are other problems, too. Want to get a good raise? You'd better publish a lot. Note that I didn't say you'd better publish excellent work. No one would say that excellent work isn't valued -- it is -- but what you really want is good numbers, because numbers are easier to evaluate. And we love indices we can point to, that can help us evaluate each other as algorithmically as possible. So each individual scientist has an h-index associated with their name (Google Scholar thinks mine is 44). Journals come with impact factors. And all of these indices are relatively easy to game, so professional advancement and stability orients itself toward gaming the indices at least as much as doing high quality work. In the meantime, a profession -- a passion, and even an art, really -- can gradually transform into a cynical race for money and prestige. And though a scientist may well grow skilled at reeling in the money during their career, whatever level of prestige they attain will ultimately fail them. As John Cacioppo argued in a previous episode of this very podcast, you and your specific work are not likely to be remembered for long, if at all. Prestige and recognition are understandable but ultimately foolish goals. Far better, Cacioppo argued, to focus your attention on the process -- on the doing of your work. And your best shot at enjoying that work -- perhaps at enjoying your life -- is to make sure that the work that you do is aligned with your values. Brian Nosek and I are in full agreement on at least one point: The system within which science is done -- particularly within which American science is done -- discourages a process-oriented focus, and, by extension, discourages us from aligning our scientific process with our values. Why? Because our institutions have to keep the lights on. So, science isn't broken at all. How could it be? Science is a system, a philosophy, perhaps even a moral commitment...to transparency and openness, to verifiability, to repeatability, to discovery, and, I would argue, to humility. Science is far more than a collection of methods and techniques, and, by the way, there is nothing about science that requires coverage by the New York Times to be valid. What may be broken is the system within which science manifests as a profession. So here's why I admire Brian Nosek so much: He isn't just complaining about things, the way I do. Instead, he's working hard to develop an alternative system -- a system based on the scientific process instead of rewarding outcomes, and, by extension, a scientific process based on deeply held scientific values. You and I may not agree with all the details in Brian's approach, but, you know, it's easy to criticize, right? Anyway, here are Brian Nosek and me, having a conversation in one of the conference rooms at the Center for Open Science.
Music for this episode of Circle of Willis was written and performed by Tom Stauffer of Tucson, Arizona. For information about how to purchase Tom’s music, as well as the music of his band THE NEW DRAKES, visit his Amazon page.
Circle of Willis is Produced by Siva Vaidhyanathan and brought brought to you by VQR and the Center for Media and Citizenship. Plus, we're a member of the TEEJ.FM podcast network. Special thanks to VQR Editor Paul Reyes, WTJU FM General Manager Nathan Moore, as well as NPR reporter and co-founder of the very popular podcast Invisibilia, Lulu Miller.
-
Jim Coan
From VQR and the Center for Media and Citizenship, this is Circle of Willis. For this episode, I talk with Brian Noseck about his efforts to increase the public's trust in scientific findings through openness, repeatability, and taking a critical look at how scientists are rewarded for their work.
Jim Coan
Everyone, welcome to my podcast Circle of Willis. For this episode, I'm sharing a conversation I had a while ago with Brian Nosek, Professor of Psychology here with me at the University of Virginia, as well as co founder and director of the Center for Open Science. Also here in frought, little Charlottesville. Brian earned his PhD at Yale University way back, way back in 2002, only about a year or so before I first met him here, when I was just a jittery job candidate. Um, Brian has been in the public eye quite a lot in the past, for many years now. Not only due to his work with the Implicit Association Test, otherwise known as the IAT, but also and perhaps mainly, for his more recent pathbreaking efforts to increase the transparency and reproducibility of the work scientists do. I think you'll find that in our conversation, Brian is relentlessly thoughtful about everything that comes up. And I want to say here publicly, that I think he's absolutely right, at the very least, about the toxicity of the current system of incentives and rewards faced by academic scientists. Occasionally, you'll hear that science is broken. It's a great sort of clickbait-ty phrase that thrives in our current social media ecosystem. But it's completely wrong, backward even. Because science is not and has never been broken. Even now, science is our most precious, life affirming, life saving human activity. Literally nothing humans have invented has done more than science to improve our welfare, to increase our sensitivity to the natural world, or to reveal the forces and mechanisms that form and constrain our miraculous universe. But the institutional structures within which science is done, they're in bad shape. At the foundation of all this, public funding for science is dismal. And that problem is yoked to the steadily declining public commitment to higher education in general. Our institutions have come to rely on bloated federal grants just to keep the lights on. And the responsibility for securing those federal dollars has fallen heavily on the shoulders of scientists who ought to be focused on making discoveries in solving the world's problems. And because that's a heavy burden institutional structures have formed to incentivize, some might say coerce scientists into striving for those federal dollars. Want to get tenure? Better, bring in some big federal grants. Want 12 months of continuous salary? Better bring in some big federal grants. You get the idea. But there are other problems, too. One a good good raise? Well, you better publish a lot. Note that I didn't say you'd better publish excellent work. No one would say that excellent work isn't valued, it is! But what you really want is numbers, because numbers are easier to evaluate. And, you know, we love indices we can point to, that can help us evaluate each other, as algorithmically as possible. So each individual scientist has, for example, an H index associated with their name. If you don't know already, what that is. just don',t I don't want to explain it right now. Just Google it "H index." Journals come with impact factors. And all these indices are relatively easy to game. So professional advancement and stability oriented self toward gaming the indices, at least as much as doing high quality work. And in the meantime, a profession, a passion, you might say, in even an art really can gradually transform into a cynical race for money and prestige. And those scientists may well grow skilled at reeling in the money during their career, whatever level of prestige they attain will ultimately fail them. So you know, so much for that. As John Cacioppo argued in a previous episode of this very podcast, you and your specific work are not likely to be remembered for long, if at all. Prestige and recognition are understandable, but ultimately pretty foolish goals. Far better, Cacioppo argued, to focus your attention on the process on the doing of your work. And your best shot at enjoying that work, perhaps even enjoying your life, is to make sure that the work you do is aligned with your values. Brian Nosek and I are in full agreement on at least one point: the system within which science is done, particularly within which American science is done, discourages that process oriented focus, and by extension discourages us from aligning our scientific process with our values. Why? Because our institutions have to keep the lights on! So yeah, science isn't broken at all. How could it be? Science is a system, a philosophy, perhaps even a moral commitment to transparency and openness, to verifiability, to repeatability to discovery. And I would argue to humility. Science is far more than a collection of methods and techniques, and by the way, there is nothing about science that requires coverage by the New York Times to be valid. What's broken is the system within which science manifests as a profession. And that gets to why I admire Brian Nosek so much. He isn't just complaining about things, you know, the way that I do. He's working hard to develop an alternative system, a system based on the scientific process, instead of rewarding outcomes. And by extension, a scientific process based on deeply held scientific values. You and I may not agree with all the details of Brian's approach, but you know, it's easy to criticize, right? Anyway, enough of this. Here's Brian Nozick and me having a conversation in one of the conference rooms at the Center for Open Science.
Jim Coan
This place is fancy, man. It's fancy.
Brian Nosek
Yeah, it's really come together.
Jim Coan
Yeah.
Brian Nosek
Well, it's a good setting for-
Jim Coan
How long have you been doing this now?
Brian Nosek
2013 in March.
Jim Coan
Wow.
Brian Nosek
We got our funding.
Jim Coan
And did you rent this place right away?
Brian Nosek
We moved into the building in June. And then over here in August, I think it was? Maybe it was September.
Jim Coan
Nice.
Brian Nosek
Might have been later than that.
Jim Coan
Yeah, it starts to blur doesn't it? Right? It's like having kids.
Brian Nosek
It was in 2013 still, but it took a while to actually get the space. And of course, we had half the space. And then, now a year and a half ago, we-
Jim Coan
You got two kitchens!
Brian Nosek
-doubled the space and now there's two kitchens.
Jim Coan
There's two kitchens. There's the Keurig. And there's the regular coffee machine.
Brian Nosek
This is the Global South. You get a Keurig, that's what I hear.
Jim Coan
What's what is the COS? I know, it's called the Center for Open Science, but how would you, you know, how would you explain it to my mom, if you could? And by the way, backstory, she doesn't have an advanced degree.
Brian Nosek
My mom nobel laureate-
Jim Coan
Or even a college degree.
Brian Nosek
I wouldn't explain it to her. The, it is an organization trying to improve science. And it is a mission driven organization that thinks about "how is it that we can make science be more like we imagined science to be?" Right? So that-
Jim Coan
Which is what? What do we imagine science to be?
Brian Nosek
Well, what we learn as a kid, that scientists are curious, they try to figure stuff out. They do little experiments of some kind on whatever they're studying.
Jim Coan
Yeah.
Brian Nosek
And then they share what they-what they found with other people.
Jim Coan
Yeah.
Brian Nosek
And then other people say, "Oh, it's kind of interesting. What about this? Or what about that?" And then they share what they figured out.
Jim Coan
And they tinker.
Brian Nosek
And yeah, it's a lot of tinkering, discovery, a lot of false starts a lot of broken beakers and explosions, and right? Of getting it wrong lots of time-
Jim Coan
Lots of times, and that's the rule.
Brian Nosek
Yeah. And at some points are saying, "Oh, I think maybe this?" And then other people say, "Oh, maybe I don't know, but you know."
Jim Coan
Obviously, we're both scientists, so this is this is sort of the sandbox that that we've learned to play in, right? And it's, but but there is this other view of what science is, which is that, you know, some really amazingly smart people. I don't know why that makes me laugh!
Brian Nosek
You're thinking of all your amazingly smart people that you self described,
Jim Coan
Right, right. Yeah. Really smart people go in into a place. That's, you know, got gothic spires. And we discover the truth, and then we come out with our, you know, very fancy looking beanies and robes on and we pronounce what the truth is.
Brian Nosek
Yeah.
Jim Coan
And-and they're, they can then take that truth and stop eating eggs to lower their cholesterol or whatever.
Brian Nosek
Right.
Jim Coan
Right? I mean, I think that's really what people think this is about.
Brian Nosek
Yeah, in some ways. But I also think there is that enjoyment of like the fifth grade science fair, right? Of an appreciation that yeah, there is that stereotype of the people at a distance.
Jim Coan
Yeah.
Brian Nosek
That are elevated somehow or separate somehow, but also the sort of boots on the ground down and dirty of just stumbling through, trying to figure out hard problems. And I think both of those are true at the same time of the imagery of, of scientists or how science works.
Jim Coan
Yeah.
Brian Nosek
But the, you know, the goal that we have is the core of that. Whether it's the elevated kind or the the sort of ordinary kind of science. The core of that is, it's a conversation, it's a search, trying to figure stuff out, and people sharing what they-what they learn with others. And people challenging that, being skeptical, right? Everyone understands it sciences about skepticism.
Jim Coan
Yeah.
Brian Nosek
That sort of thing. I don't know, I don't know, maybe this maybe that. And that's what COS exists to try to help facilitate.
Jim Coan
Right.
Brian Nosek
Is the actual conversation, right? And what does it mean conversation? How do you translate that into words that we use a lot? Well, it's openness. It's a transparent discussion of "what are we trying to figure out? How are we trying to figure it out? What did we find? What do you think?"
Jim Coan
Yeah.
Brian Nosek
That's it.
Jim Coan
I see. So the center it sounds like has a pretty broad overall mission.
Brian Nosek
Yeah.
Jim Coan
But within the center, you have the sort of open science framework.
Brian Nosek
Yeah.
Jim Coan
Right? So-so how do you distinguish that from the from the COS broadly?
Brian Nosek
Yeah. So the mission is that global concept, right? That abstract concept of just open sharing of information, facilitating the dialogue of what science is supposed to be. The how we get there is the strategy of the of the center. And that is thinking about, what are the realities facing how we get science to be that way?
Jim Coan
Yeah.
Brian Nosek
Right? And part of that is assessing what's the current culture. The culture of science has particular things that happen, that aren't quite aligned with that, that ideal-
Jim Coan
That ideal. Like what? So what are what are some of the problems with the current culture?
Brian Nosek
So what are some of the problems? Well, what is it that I do on a day to day basis? As a scientist?
Jim Coan
Yeah.
Brian Nosek
I am curious, I am trying to figure things out, I am talking to people. But it all gets grounded in this very specific system of incentives and rewards. What-what makes it so I can continue to be a scientist?
Jim Coan
Right.
Brian Nosek
That the students in my lab-
Jim Coan
How can I keep doing my work?
Brian Nosek
-can become scientists in a way that is a profession, rather than just an activity.
Jim Coan
Right.
Brian Nosek
And it's that profession part, of being a scientist, where we get this gap between what we idealize as science, and what we actually do is science.
Jim Coan
Right.
Brian Nosek
The profession part requires me to write down things that I figure out and have it go through a pipeline of people saying whether it's good enough for other people to read. And it's only when it gets through that pipeline, that I get the rewards that actually mean something for me and a career right. What is that? Publication-
Jim Coan
Well, right publication. So the end of the pipeline being really other people reading it?
Brian Nosek
Yeah. And of course, that is still aligned with the interests of science, right? Is that other people will respond and things that are impactful and important and good ideas or bad ideas that are at least worth responding to will have a conversation.
Jim Coan
Yeah.
Brian Nosek
But it gets channeled through a very specific process, which is I need to write papers, and get those papers published, in order for me to get rewards. And the goal could be, you need them to get published in order for other people to read it. That's great. That's still part of what science is supposed to be.
Jim Coan
Yeah.
Brian Nosek
That's not really what a reward is as a profession.
Jim Coan
Right.
Brian Nosek
For the profession. My reward is just getting it published.
Jim Coan
And then it's more of a question of how many.
Unknown Speaker
How many and not, are they in the right place? Right?
Jim Coan
Oh, jeez. What is the right place? The right journals? Is it in science? Is it in nature?
Brian Nosek
Right. The one word journals are the place to be, right?
Jim Coan
Oh yeah.
Brian Nosek
And some of that is inevitable, right? You have to have, there's lots of people working on things. There aren't an infinite number of jobs to be had. And so you need some system of deciding who-
Jim Coan
Who gets to do it.
Brian Nosek
-who gets advanced in their careers.
Jim Coan
Right.
Brian Nosek
And so evaluation is part of the system, inevitably. We're not going to get away.
Jim Coan
Yeah.
Brian Nosek
But how is it that we have gotten to a system where the rewards are no longer "Am I advancing the conversation in science?" And really now are focused on "Am I getting publications in the right places? Am I getting grants in order? From the right sources with the right amounts of money with the right things to support my institution?" And the the real challenge there is that these rewards of publication and grants are very concrete, very immediate.
Jim Coan
Yeah.
Brian Nosek
I know if I've achieved those. And my institution knows if I've achieved those and other people in my field know if I've achieved those, that concreteness-
Jim Coan
Yeah.
Brian Nosek
Makes them-
Jim Coan
It's nice.
Brian Nosek
Focal.
Brian Nosek
Yeah.
Brian Nosek
It makes it hard to ignore counting publications, counting where they are, counting dollars, counting how much in order to assess my worth, as a scientist.
Jim Coan
Right.
Brian Nosek
Right. Where are those other things advancing the conversation, making people think different things right? You know, pushing up people's boundaries, challenging conventional ideas. Those are abstract.
Jim Coan
Yeah.
Brian Nosek
It's not so easy to decide, have you done that?
Jim Coan
Right.
Brian Nosek
And so as a consequence, it's hard to use that as a mechanism of reward.
Jim Coan
And so maybe the consequence of the way this all is laid out, for professional advancement, is that I start honing my skill in the direction of getting sufficient numbers of publications and acquiring sufficient grant money-
Brian Nosek
Right.
Jim Coan
-And that becomes the end.
Brian Nosek
Yeah.
Jim Coan
Or at least one of the ends.
Brian Nosek
Right. It's very hard to avoid, right? So it has a couple of selection pressures. One is it selects for people that are good at writing papers-
Jim Coan
Getting papers out.
Brian Nosek
-and grants.
Jim Coan
Right.
Brian Nosek
So it starts to change who it is that gets advanced, because of who has those skills that aren't necessarily the same as seeing the conversation skills. And then the other is it reinforces us to actually focus on that as the goal. Right? So so many times, I'm sure you confront this as well, is that we get into our lab meeting discussions where we start totally excited just about the idea.
Jim Coan
Yep.
Brian Nosek
Right? Debate "Oh, my gosh, would that be so interesting? Or what others? Oh, look at that crazy piece of evidence, that doesn't make any sense." Oh, no, we have the best meeting. And in the last 15 minutes to say, "okay, so how we're going to write this up? Where are we going to send it?"
Jim Coan
I'm sure you've, you've had the experience working with graduate students, where you, you know, the arc of the graduate student experiences, you know, sort of bright, shiny curiosity and interest that steadily erodes as they go through years of looking at their CV and wondering if the count is sufficiently high. And, and that sort of thing, you know? And it is not, it's certainly not perfectly correlated, and might be slightly correlated with the level of creativity that they bring to a specific problem.
Brian Nosek
Yeah, exactly. Right. Because what scientist gets into the field, because they said, "you know, what, I like writing papers and grants." That's, that's not a thing.
Jim Coan
No, that just makes me cry.
Brian Nosek
Yeah. And of course, we do those things, but no one gets into it to do those things.
Jim Coan
No, no.
Brian Nosek
And so that's the real challenge is that because the system of rewards is focused on those things-
Jim Coan
Yeah.
Brian Nosek
-that we start to sort of drift away from why we are scientists in the first place.
Jim Coan
Okay but here's the-here's the critical question, then, for me is has that really done damage to the conversation? I mean, sure, you know, it's depressing. It's kind of a drag, right? You know-
Brian Nosek
Everybody is miserable.
Jim Coan
We need to at least be making progress. There's, there's, there's a great quote, from I can't remember which, which essay by Peter Medawar, saying something like, you know, you know, "all of these terrible things about science and scientists are 100% True."
Brian Nosek
Yeah.
Jim Coan
"And somehow that still hasn't kept science from becoming the most successful human endeavor in history."
Brian Nosek
Right.
Jim Coan
So where do we stand now?
Brian Nosek
Yeah. And I think I totally agree with the comment, right? It is the most successful endeavor in human history to figure out how things work, to advance my virtually all-
Jim Coan
Medicine-
Brian Nosek
-technological progress, you know, can be a tribute-
Jim Coan
Right
Brian Nosek
-to science. So for me, it's not a question of is, is it broken? You know, it's not working. It's more of can we do this better? And, to me, the, the answer is an obvious yes, right? I think we are much less efficient in terms of the pace of discovery and the enjoyment, frankly, of discovery. Then, we could we could improve that dramatically. And the core reasons, I think, are that because that system of reward is focused on publishing as frequently as possible in the most prestigious places possible, it pulls on strands of basic fallibilities in human reasoning, as a scientist to doing science that lead us away from the most efficient means of discovery. If there are certain things that are more publishable than other things, right? That's the reward I need.
Jim Coan
Yes.
Brian Nosek
Gotta be some basis of reward, I gotta count something. If there are certain things that are more likely to be rewarded, then I'm going to make use of these enormous frontal lobes in order to make sure I get as many of those things as possible.
Jim Coan
Yep.
Brian Nosek
Right? And so I need positive results more than negative results, I need novel results more than...
Jim Coan
You need to show up in the New York Times, you need to have a lot of productivity you can point to.
Brian Nosek
And-and it's got to be exciting and sexy and interesting.
Jim Coan
Yeah.
Brian Nosek
And of course, who doesn't want those things? Of course I want those things.
Jim Coan
Yeah.
Brian Nosek
But if those are the only things I'm rewarded for, right? If I can't publish my negative results, if I can't publish a replication or a way to try to verify or increase the precision of existing estimates. If I can't publish things that don't have a neat and tidy story, where it all wraps up together, then I am faced with a dilemma in the lab.
Jim Coan
Yeah, competing demands.
Brian Nosek
Right. What's happening in the lab, when I'm studying those hard problems that keep me up at night, they get me all excited that drive us to get into science in the first place.
Jim Coan
Yeah.
Brian Nosek
When I'm actually studying those, there's all kinds of false starts.
Jim Coan
Yes.
Brian Nosek
There's all kinds of mess, all kinds of things don't make sense. Right? That's, that's the joy of doing the science in the first place is that slow paced toward actual discovery.
Jim Coan
Yeah, sure, sure.
Brian Nosek
But the incentives are all about "nope, gotta be clean, tidy, and you got to do more of it."
Jim Coan
Yeah.
Brian Nosek
And so now I have this situation of what's really happening in the lab is kind of messy. All the incentives are for beauty. And so what do I do? There's a lot of flexibility in how science gets done. I have many studies that I do, and only a subset get written up. I have many ways to analyze my data, only a subset might get into the paper. All of that flexibility then creates opportunity for me to leverage making beauty out of mush. And I don't need to, you know, and that can be seen as very cynical like, "oh, gosh, well, Brian is just a dishonest guy, because he's going to make things look beautiful, that aren't." The challenge is if that if it was if we were really good rational thinkers, then I would know when I'm doing that, right? I would see that wait a second, I am going down this path because it looks better.
Jim Coan
It's a little bit like... a little bit like playing 20 questions with your data you're saying. I'm sort of like, you know, sooner or later you arrive at an answer that looks publishable-
Brian Nosek
It looks really good. But also, along the way, have instead of cynically just saying "I need a beautiful results to be published-"
Jim Coan
You've persuaded yourself.
Brian Nosek
I've persuaded myself.
Jim Coan
Yeah.
Brian Nosek
Right, I'm, I'm interacting with my data in an honest way.
Jim Coan
Yeah.
Brian Nosek
That is simultaneously a biased way. Right? So I am genuinely trying to figure stuff out. But because there's certain things I need, I've got motivated reasoning. I will find that nicer looking result to be more compelling.
Jim Coan
Yeah.
Brian Nosek
Right? And that study where we did as a follow up to just make sure that we had that phenomenon and it turned out to be a no, I look at it's oh my gosh, this is terrible design!
Jim Coan
Of course, we did it wrong. Yeah.
Brian Nosek
Get rid of that one. How do we even think of doing that stay? That's crazy.
Jim Coan
Yeah.
Brian Nosek
Right? It's so easy. I mean, these are things that that we know from our psychology that we do so easily and so pervasively.
Jim Coan
Yeah.
Brian Nosek
Confirmation bias.
Jim Coan
Yes.
Brian Nosek
We look, we're just, it's natural, in that general sense for us to look for information consistent with our existing beliefs rather than inconsistent.
Jim Coan
Right, right.
Brian Nosek
And we'll selectively appreciate the world aligned with that. Hindsight bias.
Jim Coan
Yeah.
Brian Nosek
We will see different outcomes and then we'll say, "Oh, of course, we would have thought that this would only work with men and not with women. Yeah. That's, that's what my hypothesis would have been."
Jim Coan
Yeah.
Brian Nosek
And I would have- I even remember it as that was my hypothesis, even though I hadn't even thought about gender initially. Right? So we just do these things so fluidly, that we don't even see that we're doing them.
Jim Coan
Right.
Brian Nosek
And that's a real challenge for, for some, human doing science.
Jim Coan
And just bring us full circle a little bit. Part of the problem with the incentive structure you're arguing is that it doesn't encourage us to worry about that.
Brian Nosek
Exactly.
Jim Coan
It encourages us to take advantage of it then rather than be afraid of it.
Brian Nosek
Right. And that's, that's really the core of we're going back to this idea of how we idealize science to be as this open conversation of "this is what I did and this is what I found."
Jim Coan
Yeah.
Brian Nosek
Right? We are- the process is not transparent. What you get when you read one of my papers is my summary of what I think I did.
Jim Coan
After it's gone through a potentially very complex process.
Brian Nosek
Right. And-and it is my most genuine attempt at being accurate.
Jim Coan
This is how we are.
Brian Nosek
Yeah, this is what it is "we" the big "we."
Jim Coan
Yeah.
Brian Nosek
And so if there's fingers to point than we have to point them everywhere.
Jim Coan
Yeah.
Brian Nosek
And of course there are people that deliberately take advantage of science.
Jim Coan
Right.
Brian Nosek
And I put that aside-
Jim Coan
And that's sort of a special category I think.
Brian Nosek
It is and you know, it's nice to sort of say, "oh, that's, that's qualitatively distinct." And-and in some cases, it certainly is. But and there is a continuum, right? People are more or less willing to sort of pay attention to the fact that they might be biased. Nevertheless, I don't care about that, right?
Jim Coan
Yeah.
Brian Nosek
In the specific sense of, there will be cheaters in any system.
Jim Coan
Always.
Brian Nosek
Always.
Jim Coan
Yeah.
Brian Nosek
And we have to worry about that as a different phenomenon-
Jim Coan
Yeah.
Brian Nosek
-than the ordinariness of human reasoning-
Jim Coan
Yeah.
Brian Nosek
-trying to do science. So I'm worried about myself, in the sense of, "I want to do the best science I can. I want the results to be as credible as I can."
Jim Coan
Yeah.
Brian Nosek
And I want to provide solutions that work for me. And ideally, will work for other people, too, because in some ways, they're like me, just trying to figure it all out.
Jim Coan
We're just trying to figure it all out. We're all... we're all best intentioned people trying to do the best work that we can but yoked to these incentives that we can't do a heck of a lot about at this.
Brian Nosek
Yeah. That's right. And until you can actually see my process, right? Of how it is I get to those... those conclusions at the end, then there isn't really opportunity to do what science is supposedly able to do, self correct. That's really where this core challenge is, is that because all you see in the- in the current process, all you see is my summary report of what I did. Then you are reliant in terms of an independent observer, you are reliant on what I am able to and did report.
Jim Coan
Yeah.
Brian Nosek
To try to get insight-
Jim Coan
Yeah.
Brian Nosek
-where it might be flawed? And where it might be good.
Jim Coan
So this is the COS trying to open up...Okay, this gets back to that question I asked, I started to deflect this from the Open Science Framework as sort of underneath the overall umbrella of the COS mission, the Open Science Framework is about sharing the process.
Brian Nosek
That's right. Yeah.
Jim Coan
Not just a summary.
Brian Nosek
Exactly. How do we open up the entire research lifecycle?
Jim Coan
But, but... that scares the shit out of me!
Brian Nosek
Yeah?
Jim Coan
Yeah, a little bit.
Brian Nosek
How so?
Jim Coan
Because... because my process is a complete mess! I mean, I don't want anybody to see my process. It's like, it's like walking around... That's like coming to work naked.
Brian Nosek
Yeah.
Jim Coan
You know? I mean...
Brian Nosek
Yep. Well, have you ever been to a nudist camp actually is quite freeing. And not that I've ever been, but I it-
Jim Coan
You read about it in a magazine.
Brian Nosek
I read about it a lot. Yeah. Oh, wait, no, no.
Jim Coan
Seriously though. I mean, you know, this, this seems to be this is a totally different view. I mean, and I guess that that also gets us back to the the incentive structure? I mean, you know, after gosh, I mean, in one way or another, I start doing research as an undergrad. Right? So now we're going on 25 years.
Brian Nosek
Yeah.
Jim Coan
I've been thinking in terms of, you know, presenting the narrative, you know?
Brian Nosek
Exactly.
Jim Coan
Getting, you know, you know, thinking about process, what in the hell are you talking about? My process is a, it's like, it's like a bunch of spaghetti noodles all over the table.
Brian Nosek
Right. Yeah. And it is. It is difficult to imagine exposing the entire thing. And so there are two answers, I guess to that one is incrementalism. What can-what else can you expose confidently? Do that.
Jim Coan
Yeah.
Brian Nosek
And so the the Open Science Framework as infrastructure, as software, gives researchers full flexibility. You can leave whatever private you want, you can expose whatever you want. And we make it easy to make it more accessible. Right? So that's the first thing is meet researchers where they are. They want to-
Jim Coan
So anybody can-anybody can sign up for this? Anybody can-
Brian Nosek
It's free service. It's for you know, it's basically for archiving for your own use, right? We should stop losing our own data, our own materials for our own labs use.
Jim Coan
Yeah.
Brian Nosek
That's, it's ridiculous that when a machine blows up in our own lab or a grad student leaves, that we lose information.
Jim Coan
Yeah.
Brian Nosek
For our own use. That's crazy.
Jim Coan
Yeah, that is crazy yeah.
Brian Nosek
Allow people to use the system entirely privately and make it really easy to open. So that's one answer to that, which is, "okay, you're not ready to go totally open, fine. Don't go totally open." But you can still get the benefits of preservation of making it possible to make parts of it more accessible, and of having more confidence internally of your own process.
Jim Coan
Right.
Brian Nosek
Even if you're not willing to expose it.
Jim Coan
Right, right, right.
Brian Nosek
That's one part. The other part is that it does sort of prompt a mind shift in... if I, if I am so alarmed to let people see behind the curtain of my paper. What is that? Does that mean something about the confidence that I have in what's actually goes into that paper? Right? It is partly it's like, well, "I just didn't document it right." So part of your answer, I assume would be like mine, which is, yeah, it's a mess, but it's not that it's an inaccurate mess.
Jim Coan
It's just- I'm a bad bookkeeper.
Brian Nosek
Right, right.
Jim Coan
Yeah. I'm a terrible bookkeeper.
Brian Nosek
And the outputs, totally right. I'm sure. It's totally right.
Jim Coan
Yeah.
Brian Nosek
But of course, you know that there is overconfidence in that right? Sometimes-
Jim Coan
Yeah.
Brian Nosek
-I'll go back to these old files that I had in these old thing, you know, the reviews come back a year later, or, you know, a project comes up and I say, "oh, there's this old project we have, let's go check it out."
Jim Coan
Right.
Brian Nosek
And, and I discover all kinds of errors, or I can't figure it out. It's like, oh, jeez.
Jim Coan
You know, I got to tell you. One time, we, we reanalyze, we this is this is an ongoing project. There's something just looks too good to be true in a dataset. So I asked another person in the lab to reanalyze it. Just however, they would like to do it, and got a totally different answer. It's not even doing a different study. It's the same exact data set.
Brian Nosek
Yeah.
Jim Coan
And-and I just, I just wandered around like I'd been hit in the head for days.
Brian Nosek
Right. And so this is the real challenge, I think, is that because we don't ever anticipate that the process will be public, we don't care for the process and the way that we might otherwise.
Jim Coan
Yeah.
Brian Nosek
So if we know in advance, right? That I- in this my lab is moving step by step this direction is: we're going to make as much of our process accessible as possible. Knowing that we're going to do that changes how we do our process, right? So-
Jim Coan
Yeah.
Brian Nosek
-it isn't those spaghetti noodles, we actually think about it in the same way that we've got about a paper. Which is, yes, it's gonna take a while to get it to all be aligned and look right. But that's part of doing the science.
Jim Coan
So you put a little more effort into sort of formalizing the process is kind of what you're saying.
Brian Nosek
Yeah, so we have a, you know-
Jim Coan
The way that you do the narrative at the end.
Brian Nosek
Exactly. Right. So it's not just the narrative at the end. Now it's the codebook for the data.
Jim Coan
Yeah.
Brian Nosek
Now it's the code that's applied to the data.
Jim Coan
Right.
Brian Nosek
Now it's the, you know, the materials that we use.
Jim Coan
The hypotheses.
Brian Nosek
The hypotheses that we say, yeah, we just, we write it down in advance. And there is still the spaghetti mess, but it is, it leads to an actual set of these... this is the idea at the beginning. These are the materials that we have these the data we have, so that all of that is clean, and beautiful. And I can go back to it. I mean, it's amazing now to go back to a study that we've done in the last two years, compared to a study we did six years ago and actually be able to understand it, actually. And say-
Jim Coan
Well, in theory, anybody could? You could reconstruct-
Brian Nosek
Right, you can say, there's a few of our recent papers where you can go to the OSF, find the code, click run, and regenerate all of our findings from the paper. Just right-
Jim Coan
Shit.
Brian Nosek
-instantly. That's amazing. Right? That someone else can confirm that we got what we say we got.
Jim Coan
It sounds better than a method section.
Brian Nosek
Yeah. Right.
Jim Coan
So this is all-this is all part of the, there's a conversation that happened a few years ago, is, should the method section be sufficient? My own feeling is that maybe it should, but it's never going to be.
Brian Nosek
No, exactly. And-and it's, if we focus on the paper, we're going to end up missing the boat entirely.
Jim Coan
A whole bunch of stuff.
Brian Nosek
Right? The sum, the methods section is a summary. It's a summary in order to facilitate the purpose of a paper.
Jim Coan
Right.
Brian Nosek
Which is to give a reader, to understand-
Jim Coan
A basic understanding of what happened, yeah.
Brian Nosek
But the reader that actually needs to know precisely what happened, cannot have to depend on the paper.
Jim Coan
Yeah.
Brian Nosek
Right? They have to be able to get into the materials. A video of the experimental setup.
Jim Coan
Yeah.
Brian Nosek
Whatever it is, there's lots of other stuff that when I actually want to take what you did, and apply it in some way, in our lab, we need to be able to have a much deeper engagement with that.
Jim Coan
And you know, it seems like if you go back in the history of science, this is... there's a way in which this was always the case.
Brian Nosek
Yeah.
Jim Coan
Right? I mean, it's not your-you seem to be really formalizing, or maybe that's the wrong word. You're providing a technology for accelerating or making this process easier. But when I think about the sort of major scientists in the major fields of science, I mean, visiting laboratories, conferences, you know, conversations like this, over drinks at the bar, you know, where people start working out some of these details and start learning about each other's process. And this sort of, I think, has always been right part of the-
Brian Nosek
How could it have been otherwise?
Jim Coan
How could it have been otherwise! It couldn't possibly have been otherwise. We're not mind reader's.
Brian Nosek
Yeah. And so we've lost this perspective, partly as a function of being able to get just enough information at a distance.
Jim Coan
Yeah.
Brian Nosek
The papers itself, the fact that they're distributed easily, makes it feel like we understand what each other is doing. The internet dramatically accelerates all of that.
Brian Nosek
Yeah.
Brian Nosek
Right? And simultaneously the feet... Science has gotten so big, right? So when there's only a few people, then you have relationships. And you call up the person you say, "how did you do that? Oh, my God, we tried this thing. It didn't make any sense." So those relationships facilitate that exchange of information-
Jim Coan
Yeah.
Brian Nosek
-in an easy way. Now that there are hundreds, thousands of people that I would consider to be studying the same kinds of things that I study. I can't-
Jim Coan
You can't do that.
Brian Nosek
Yeah. And so I am more reliant on the more indirect means-
Jim Coan
Yeah.
Brian Nosek
-of communication.
Jim Coan
Well, in a similar way, you know, meet... when you do meet at the at the conference, not everybody has access to that conversation that you had, you know?
Brian Nosek
Right...
Jim Coan
Really interesting, synergistic things happening. And then that brings in diversity and access and all of these other questions coming.
Brian Nosek
Yeah. So some people say, "Oh, we totally know how they did their study."
Jim Coan
"Because we talked to them."
Brian Nosek
"We know them."
Jim Coan
Yeah. Yeah. We're buddies. We went out and saw band in Austin afterwards.
Brian Nosek
Right. And then they're the 150 other people aren't in the group-
Jim Coan
Standing around fogging up the glass.
Brian Nosek
-saying. "What the heck?"
Jim Coan
How come-how come you didn't tell me that you didn't replicate that study? You know, after 10 tries? Because I did too. And now we both wasted our time.
Brian Nosek
Right, yeah. And so that is a real critical point is the accessibility of that, right? And the degree to which it creates this elitism in science, right? We are so lucky to be at a place like University of Virginia, where we have access to everybody.
Jim Coan
Yeah.
Brian Nosek
If I email people-
Jim Coan
Yeah.
Brian Nosek
-they usually respond. Because we have a position, a status, credibility that's instantly applied based on the circumstances that we're in. And that it's, it's easy to not see that privilege, in the insight that we have in our own domains.
Jim Coan
And to-and to also not see the opportunity cost that comes along with it. Because- for science in general.
Brian Nosek
Yeah.
Jim Coan
Because we're pretty smart, but we're not that smart. There's all kinds of people out there in the world who don't have this access that could be really contributing in important ways to the problems, we're wrestling, right. And that currently don't have access to that.
Brian Nosek
Right, yeah. And so that really is the... It's not subversive, because it's, it's stated. What the Open Science Framework can do is really provide inclusivity at scale, right?
Jim Coan
Yeah.
Brian Nosek
If you and I are getting insight on each other's work based on our ability to have conversation and our ability to share materials directly, email with each other, whatever. By putting that into an open repository. Now we give an orders of magnitude, more people, the opportunity to get similar kinds of insight.
Jim Coan
Right.
Brian Nosek
Now, of course that personal contact may always have a benefit, right? There is those networks, social networks really matter. But we can do a lot to help the spread of that information, accelerate and be much broader so that people that don't have the same access points can get started.
Jim Coan
Right.
Brian Nosek
And get into the system, can get enough insight because of that to greatly accelerate their own capacity, but then the capacity of science more broadly.
Jim Coan
So this sounds so great. Right? This sounds really great, but it also sounds... I mean, the thing about going to the conference, is that it's kind of easy.
Brian Nosek
Yeah.
Jim Coan
You know? You can you sidle up to the bar, and you have a beer and you-
Brian Nosek
And it's fun. Yeah.
Jim Coan
And it's fun. And, you know, making the process more transparent sounds hard. Is it hard?
Brian Nosek
It is hard in the sense that we, none of us have done it before, in a systematic way.
Jim Coan
Yeah.
Brian Nosek
And so, in the same way that it's hard to write a paper, or it's hard to do data analysis, of course, it's you got to learn.
Jim Coan
Yeah.
Brian Nosek
All of these things have to be learned.
Jim Coan
Shit, I don't want to learn anything. I don't know a whole bunch of stuff so far.
Brian Nosek
But you're always learning right? This is the thing that we forget as scientists is that we study things that we don't we are studying them to learn about that.
Jim Coan
Yeah.
Brian Nosek
And the process part can feel boring, right?
Jim Coan
Yeah.
Brian Nosek
How do we format our papers? I don't care how we format just tell me to format the damn paper.
Jim Coan
Right.
Brian Nosek
And once I've learned that, don't tell me again, I know APA style now. I am not learning any other style. Forget it. I don't care.
Jim Coan
Nor am I learning the update.
Brian Nosek
Yeah, right. Yeah. I'm all APA fourth edition. That's that's where I ended.
Jim Coan
Right.
Brian Nosek
But those, but you know, so that that's going to be a barrier. Learning new stuff when you feel like you already know it or have a system or workflow, is a barrier to change. But once it's learned, it's not more time. It's just this is how things are done. This is how we do it. This is how we document and there is long term efficiency benefits for doing a good job of documenting your process, of organizing your process because of all of those revisits that we do.
Jim Coan
Yeah.
Brian Nosek
Right? How many times has it happened in my lab, I don't know how many times it's happened in yours, it's I happen a lot in my lab where we are having a conversation, and we say, "this is just like this other thing that we did with Nicole five years ago. And let's, let's go back to those data." And that that study does-
Jim Coan
Reanalyze it.
Brian Nosek
Reanalyze it, take it uses a pilot for this new thing. And we just can't.
Jim Coan
Yeah.
Brian Nosek
We've lost the thread of that information for own use. And we've just got, it's, we're stumbling over ourselves, because of our lack of good training in how to curate and manage-
Jim Coan
The process.
Brian Nosek
-our own process for our own use. And so that's where I think we can ultimately make a case that it's worth people's time to-to learn that stuff. Because it's not just for the greater good, so that someone else that they don't know can learn about this. It's so that themselves a year from now, can get right back into it.
Jim Coan
Yeah.
Brian Nosek
And keep moving.
Jim Coan
Yeah, you know, you do - I don't know if you've had this experience, I imagine you have -but-but I said, certainly, you know, I've read various histories of science. And you know, you know, scientists work and one thing that I've always really envied is the is this sort of progressive, systematic working through a problem over and over again, where you just keep building like this, this, this trajectory, you know? Rather than sort of having this cool finding, and this cool finding, and that cool finding where that sort of, you know, cumulative-
Brian Nosek
Right.
Jim Coan
-work, it seems so, so important and enviable to me.
Brian Nosek
Yeah.
Jim Coan
And that process would really, really help that.
Brian Nosek
Yeah, and historically, that incrementalism, which has been seen as a bad word in science, but it really is that cumulative nature, is building and on different findings. That historically has been a single person or a single small group working on that. But now the problems that we have are so big, that there are lots of people working on that.
Jim Coan
Yeah.
Brian Nosek
Right? Why would- why are there these huge debates about CRISPR, and who is the originator? Well, it's because lots of people were working on the same problem, whereas there aren't those same debates for a lot of historical problems, because it was that person-
Jim Coan
Yeah.
Brian Nosek
-that was working on that problem.
Jim Coan
With massive funding, or sort of, you know-
Brian Nosek
And many years.
Jim Coan
-many years-
Brian Nosek
Right.
Jim Coan
-lots of, lots of time to spend-
Brian Nosek
Yeah.
Jim Coan
-sort of sipping tea and contemplating the next move.
Brian Nosek
Yep. But now the spread of that just amplifies the need for better communication, more transparency of the process of the materials and everything else so that the many minds working on the same problem have access to the same knowledge base.
Jim Coan
Do you think it's going to slow things down?
Brian Nosek
I guess it depends on what you mean, in the sense of it does take a little bit more time to write down rather than just say, "Oh, I'll remember." And move on.
Jim Coan
Yeah.
Brian Nosek
In the short term that takes more time. In the long term, we don't just remember, and we forget, we lose things and everything else. So there's- I think the cumulative benefit is that things will move faster. But at the same time, it's not, I think, that we'll necessarily be faster in any one line of research. It may, it may be slower, it just may yield more.
Jim Coan
It may give us this sort of subjective experience of being slower.
Brian Nosek
Yeah. And simultaneously, we may get more, we may make more progress in science, may discover more things more quickly. We'll be more efficient in process.
Jim Coan
So this gets us to the crisis.
Brian Nosek
Yeah.
Jim Coan
Right? If I can use that. I mean, I don't know if that's even the right way to put it anymore. But-but-
Brian Nosek
We call it the reproducibility movement.
Jim Coan
The repro- oh, God, I was gonna ask you if the if there is a movement one, and what the name of the movement is, because it's sort of like I don't want to say the wrong thing and offend people. But at this point, I just sort of have resigned myself to offending someone. That's the social media age. But-but let's call it the reproducibility movement.
Brian Nosek
Yeah.
Jim Coan
Can we call it that?
Brian Nosek
Sure.
Jim Coan
So it seems like you know, we all have learned that there's these trade offs. There's the type one error or type two error trade off. There's the external validity, internal validity trade off. And these are things that we all understand and we all sort of pretty much know. I mean one thing, with the current state of affairs where people are just like, bam, bam, bam, bam, bam, bam, getting a bunch of papers out as fast as they can is that the trade off there is that we might make more discoveries at a at a faster rate, but a lot more of them might be wrong.
Brian Nosek
Yeah.
Jim Coan
And so the other way to look at this, you know, if we focus more on the process, we might make discoveries at a slower rate. But we might be right more of the time. Do you think that that's a fair characterization?
Brian Nosek
It is fair in the sense that is at that sort of first level of analysis, that would seem to be the trade offs. And you can make arguments that it isn't quite that way. But let's go with that.
Jim Coan
Yeah, and I'm willing to hear that. I mean, I don't know whether that's right or not.
Brian Nosek
So then the ultimate question is what is optimal? Because the, let's say that the proportion is different and the number of true discoveries - we'll use true, loosely here.
Jim Coan
Yeah.
Brian Nosek
And the total volume of discovery.
Jim Coan
Right.
Brian Nosek
Right? And so there has to be at some point where there's there's an optimization curve of what- which way you should do it. There are some parts that suggest that it isn't just as simple as a trade off between the two, the best example probably is low powered research. Right? Now, it's to my benefit to generate as many... I actually can generate more findings by using small samples.
Jim Coan
Right. And low by low power, we mean, the insufficient ability to detect a true effect.
Brian Nosek
Right, if it's actually there.
Jim Coan
Yeah, it's actually there.
Brian Nosek
And the... say, well, low power is bad for identifying, you'll miss more things than you should have. Right? You'll get false negatives, right? If I have low power, then I can't detect true effects that are there-
Jim Coan
Yeah.
Brian Nosek
-so low power is bad for negative. But if I get a positive, then it's more likely to be true. Right?
Jim Coan
Right.
Brian Nosek
And that intuition is not correct. Low power is bad for both false negatives and false positives. In the conditions of what's the rate of actually true things that we're studying, right? And so the difficulty of the intuition there is that the base rate of how many true things are there to discover strongly influences the impact of power on our rate of discovering findings. And if we're studying things that are not likely to be true, I mean, because we have lots of ideas about what's possible in this hard domain, and only-
Jim Coan
We're gonna be wrong most of the time.
Brian Nosek
We're gonna be wrong most of the time. If we accept that, then it's in our interest to do much more high powered research to increase or decrease both our false negative and our false positive rate. So that is an argument for, it isn't a strict trade off, doing fewer studies with higher power. Well, of course, then we're going to discover fewer things. No, no, we're actually going to discover more things, because we're going to have both the power to detect the true things and the power to say those other things aren't there that we didn't think are there. And so that's, that's an argument that is more complicated than a simple "Which, which, which error rate are we more interested in?"
Brian Nosek
Yeah and it reminds me that I've never yet heard a good persuasive argument for low power.
Brian Nosek
Yeah.
Jim Coan
In research, that's never happened.
Brian Nosek
Right. There is a- there isn't! It's a waste of resources.
Jim Coan
But there may be a trade off that has to do with focusing on the narrative versus the process. There may yet, because you could still because power can be a separate issue.
Brian Nosek
Yeah, so the the argument, there's a sort of a side argument that says, Well, low p- everybody can agree if it's true, that low power is bad. Nevertheless, there are occasions where low powered research is worth talking about. Right?
Jim Coan
That's true.
Brian Nosek
And that's an important qualification, which is, look, there are only 17 People with this particular disorder in the world.
Jim Coan
Absolutely.
Brian Nosek
We're going to talk about what we found. Well, those samples-
Jim Coan
That's absolutely right. But that raises the point that I think doesn't get sufficiently talked about sometimes, which is that statistical power isn't just about n, the number of subjects you have.
Brian Nosek
Yeah.
Jim Coan
Design-
Brian Nosek
Yeah.
Jim Coan
-has a huge impact on power. So you can take those 17, I mean, the development of exposure therapy-
Brian Nosek
Yeah.
Jim Coan
-which is far and away, the most effective clinical intervention ever devised by clinical science. Was built on n of one studies, over and over and over again. Because of the designs, because of the very careful way that they documented changes when-when interventions were implemented.
Brian Nosek
Yeah.
Jim Coan
Those are logical arguments. And power isn't... is a logical argument.
Brian Nosek
And even in the scenarios, right, we can imagine many cases where we have very little data, but it's such precious data.
Jim Coan
Yeah.
Brian Nosek
We want to talk about it.
Jim Coan
That's right. Yeah, that's right. Setting that aside-
Brian Nosek
So we will, we will accept the point and or embrace the point that wh-whether we take data seriously or not, has lots of different influences. But simultaneously, the confidence that we have in data is limited by statistical power... So we will just make slower progress when it's harder to get the data that we need. That's just a reality. And so that's, that's a hard thing for a lot of the in this reproducibility movement for a lot of people to face because of the high variation in how hard it is to get some kinds of data, right?
Jim Coan
Yeah.
Brian Nosek
It's really hard to get a lot of data from babies.
Jim Coan
Yeah.
Brian Nosek
It's hard to get the babies into the experiment itself. And it's hard to keep them engaged enough to get enough data from them.
Jim Coan
So I don't study babies.
Brian Nosek
And so yeah, and there that is, if you're concerned about power then studying babies is a bad idea. Because it is really, really hard.
Jim Coan
Yeah.
Brian Nosek
And this is, this is a very difficult thing to confront, right? There's an entire discipline focused on data collection with babies, and to say, "well, we just really can't do that very efficiently, or effectively" is like, what are you talking about? Are you saying we need to shut down this area of research? Well, no. But we do if... because we think that's important. We want to understand babies interior lives. We are going to have to accept, given the realities, of the statistical realities of power, that our progress is going to be much slower than in other fields. And so the number, the pace of those amazing discoveries and confidence in those discoveries, is necessarily more slow, in fields where it's hard to get sufficient data. And so the standards have to be different in order to evaluate that compared to psychophysics.
Jim Coan
Do you think that that's a general principle? Or do you think then there again, it sort of depends on what we're studying? It sort of depends on what- because I think of Judy DeLoache's work with toddlers, for example. And I did one of these with her, and she was talking about how the scale error effect.
Brian Nosek
Right.
Jim Coan
You know, they published that with no statistics at all.
Brian Nosek
Yeah.
Jim Coan
Because it was just does it happen or not?
Brian Nosek
Yes. So that's exactly-that's a fantastic point. And that is because you can study things. You're just limited to what is possible, right? If the question is existence proof?
Jim Coan
Yeah.
Brian Nosek
One person's enough.
Jim Coan
Yeah.
Brian Nosek
And one instance of it, on that one person is enough. If the question is for very large effects, great.
Jim Coan
Yeah.
Brian Nosek
Small samples, no problem. But you know, Uri Simonsohn has a great paper on the, this argument about telescopes.
Jim Coan
Yeah.
Brian Nosek
Right?
Jim Coan
Yeah, it's great.
Brian Nosek
He simply cannot study something if you don't have the precision to measure it.
Brian Nosek
Yeah.
Brian Nosek
You simply can't do it.
Jim Coan
Yeah.
Brian Nosek
And so saying, we're going to study these galaxies that are this, you know, light years away with this tiny little telescope. It's a non starter.
Jim Coan
Yeah.
Brian Nosek
And we don't, it's easy in the context of a physical telescope and seeing something to understand that, it's much harder to see that in the statistics of inference.
Jim Coan
Yeah, absolutely.
Brian Nosek
The same concept applies.
Jim Coan
Yeah.
Brian Nosek
That's a real challenge.
Jim Coan
So is then part of what the Center for Open Science does... So it provides us this opportunity to share our process, maybe some guidelines for how to do that. Is it also policing? Is it part-is it part of a sort of a methodological police force to sort of spot bad practices?
Brian Nosek
No, in the sense that the there isn't anybody with enforcement capacity in science.
Jim Coan
But do you think it would be helpful to have like a centralized? Like a regulatory body or a-
Brian Nosek
No, I'm pretty libertarian about how science operates best. And that is that what we want, I think, in a scientific, a productive scientific process is an open marketplace of ideas.
Jim Coan
Yeah.
Brian Nosek
And that open marketplace of ideas in order for it to work -and this is really where I think it's not working where it should - is that we don't actually have insight into how those ideas, what evidence is supporting those ideas. And that's the transparency part.
Jim Coan
Yeah.
Brian Nosek
So the main change that I would like to see is transparency of those ideas so that self correction of how science is supposed to operate as a distributed system where there is no authority, there is no hierarchy, there's just ideas and there's evidence, and those get debated and challenged, and there's clusters of support form, this group thinks that that's true, that group thinks that's true. Let them fight it out. Right? Does it actually yield the truth? Does that emerge from all of that system? Well, you know, that's, that's a hard problem to figure out. But I think it is the kind of system that we can get at sort of the ground level, as researchers deal with it. The systems of how do we pull information out of that open free marketplace and insert it into policy or into other human activities? Well, now that's something where you might have structures that aren't as libertarian. Where it is a matter of reputation-
Jim Coan
Because that's politics.
Brian Nosek
-decision making, right. There, there is decision making to turn those ideas, the ideas in that marketplace into actions.
Jim Coan
Right.
Brian Nosek
And that has to have some kind of process of who is deciding what are the- who are they deciding for? And so that that's, that's but that's a different kind problem.
Jim Coan
Yeah.
Brian Nosek
Then just the the activities of individual researchers trying to figure stuff out. So, getting back to what your question was, no, I don't think there's... police is not a role in that environment. But critical evaluation is.
Jim Coan
It must be.
Brian Nosek
Skepticism is.
Jim Coan
There's no science without that.
Brian Nosek
But-but it isn't with differential status, right? The implication of police force is that there are some people that are in charge.
Jim Coan
You have the truth, and other- or you know, how it's done.
Brian Nosek
Right. And we are going to judge you.
Jim Coan
Yeah.
Brian Nosek
And I don't think there's, there's... that's how it should work. How it should work, which is close to what it does now, which is you and I disagree. And we argue about it. And I say you should have done it that way. And you say "Screw you, I did it this way." And then well, that's where it ends
Jim Coan
It's really interesting because I think about, you know, you know, I did a sort of methodology minor in my PhD. And we talked a lot about philosophy of science over the years. And there's this sort of what you're doing with this doesn't align perfectly or conflict with any of the major philosophical movements that I can think of.
Brian Nosek
Yeah.
Jim Coan
It seems like, you know, if we had Karl Popper and Paul Feyeraben here, neither would be particularly offended by what you're talking about. Your qualifier- I haven't had, an epistemological anarchy, right? It's just anything goes in order to get to the truth. But I would guess that part of that would be openness about process.
Brian Nosek
Yeah, I think real - and this is really, that's an important point. Because I think really, what our exclusive aim is, is on embodying the values of the enterprise.
Jim Coan
Yeah.
Brian Nosek
Right? Not of what it is that, how one gets to truth, right? The philosophies of science can all be layered on top of sort of the basic units of you need to communicate.
Jim Coan
You need to communicate.
Brian Nosek
You have to share what you've done.
Jim Coan
Share.
Brian Nosek
You have to be open with your process.
Jim Coan
Yeah.
Brian Nosek
Right? How else are we going to evaluate it? Right? What would be the credible philosophical argument of No, nobody share anything, we will argue about who's right. Right? They're just I mean, it's sort of like-
Jim Coan
Well, that's religion, right? I mean, not to offend my religious friends. Yeah, there I did it. Cat's out of the bag.
Brian Nosek
But-but that's really what it is. Right? Is that you have to is as a precept for actually having intellectual discussion to try to figure something out, these are the basics.
Jim Coan
Yeah.
Brian Nosek
And that's all that we're trying to do is not push beyond that, of just getting the marketplace, the information it needs for those debates to happen.
Jim Coan
I love it. Brian, this was really great conversation. Thank you so much.
Brian Nosek
It was really fun.
Jim Coan
Alright.
Brian Nosek
Great to chat.
Jim Coan
Okay, folks, that's it. Thanks to Brian Nosek for sharing his time with me and for his candor, good humor and great attitude. Thanks to Brian also for the work he's done, and continues to do to bring real practical resources to bear via the Center for Open Science. To encourage and empower scientists to focus on their values as they do the difficult and deeply rewarding work that they do. Thanks again, Brian, I admire you a lot. Folks, the music for this episode of Circle of Willis was written and performed by Tom Stauffer of Tucson, Arizona. For information about how to purchase Tom's music, as well as the music of his band the New Drake's check the about page at circleofwillispodcast.com. Also check out our new Instagram account. That's right. There is an account on Instagram now called circleofwillispodcast. And there you'll find a series of essays and drawings and mini blogs about topics related to the program, all of which are only available on Instagram. Why? Because I want to! That's wh! I like it. And that's all. Circle of Willis is produced by Siva Vaidhyanathan and brought to you by VQR and the Center for Media and Citizenship here at the University of Virginia. And Circle of Willis is a member of the Teej FM Podcast Network. You can find out more about that at teej.fm (now wtju.net). Special thanks to VQR Editor Paul Reyes, WTJU FM General Manager Nathan Moore, as well as NPR reporter and co founder of the very popular podcast invisibilia, Lulu Miller. If you liked this podcast, how about giving us a little review in iTunes and letting us know how we're doing? It's super easy and we like it. Or go the more direct route by simply sending us an email at circleofwillispod@gmail.com That is circleofwillispod@gmail.com. You can also contact us by visiting circleofwillispodcast.com and clicking on the Contact tab. Okay? Okay, folks, here's the deal. I'm taking a vacation. Yes, that is what I'm going to do. That means this will be our last episode until August, when Circle of Willis will return with a vengeance. I don't really know what I mean by that. But when we do return, we'll be hearing from Wendy Hassenkamp, science director of the Mind and Life Institute to hear about what she's been doing to align the science of contemplative practice with all those processes and values that Brian Nosek and I talked about only moments ago. We'll also be hearing from John and Julie Gottman about the science of marriage in our very first episode of Circle of Willis recorded before a live audience. And there's so much more coming up too! Until then, I hope you're planning on having as nice a time this summer as I am. See you in August. Bye bye
Transcribed by https://otter.ai