8: Simine Vazire
Welcome to Episode 8, where I talk to SIMINE VAZIRE, Associate Professor of Psychology at the University of California at Davis, about the stability of personality, our ability to know ourselves, and some of the nuances within the prescriptive advice of the Open Science Movement. Simine wears a number of different hats. In recent years, she’s been at or near the center of ongoing conversations among scientists about the virtues and challenges of open science. As part of this work, she co-founded the Society for the Improvement of Psychological Science (SIPS) and co-hosts a science podcast (with Sanjay Srivastava and Alexa Tullett) called THE BLACK GOAT. Simine is also editor in chief of the journal Social Psychological and Personality Science and a senior editor at Collabra. Interestingly, Simine has also been a part of the conversation about the process of criticism in science. As most listeners well know, criticism is unquestionably essential if science is going to be self-correcting (which is of course the whole point!). One question the field has been grappling with is the point at which criticism crosses over into harassment and bullying—a question at the heart of a recent op-ed Simine wrote for Slate. I have my own thoughts on this question, which I’ll save for another time, but one of the reasons I was so keen to ask Simine to be on Circle of Willis is that I find her approach to grappling with such questions to be equal parts humble, charitable, and firm. She isn’t likely to allow a legitimate criticism to be brushed aside in order to avoid hurting someone's feelings, but neither is she going to participate in (or for that matter tolerate) bullying. I think that in our age of shoot-from-the-hip outrage, that can be a hard path to find, let alone walk, and I genuinely admire her efforts. There are many other things I love about Simine, but as you’ll hear in this episode, at or near the top of the list of her agreeable traits is that she’ll be the first to tell any of you that sometimes she’s wrong. We try to be right while tolerating (and admitting to) our mistakes. Oh, and — seriously — keep a notepad handy for this episode. Simine is unusually quotable! * * * As always, remember that this podcast is brought to you by VQR and the Center for Media and Citizenship. Plus, we're a member of the TEEJ.FM podcast network. AND... The music of CIRCLE OF WILLIS was composed and performed by Tom Stauffer, Gene Ruley and their band THE NEW DRAKES. You can purchase this music at their Amazon page.
-
Jim Coan
From VQR and the Center for Media and Citizenship, this is episode eight of Circle of Willis, where I chat with personality psychologist and open science activist Simine Vazire about the advantages and prescriptive advice of the Open Science movement.
Jim Coan
Everyone, it's Jim Coan. This is my podcast Circle of Willis. I don't know why that's funny. It's funny, it strikes me as funny right this moment. For this episode, I'm talking with Simine Vazire. Who I'm just gonna go ahead and characterize as an activist in what is increasingly being called the Open Science movement. Although it hasn't really always been about Open Science, per se and we don't really refer to it as the Open Science movement in our conversation. I think, I just like the sound of Open Science Movement to, you know, the Replication Movement or the Movement to Increase the Repeatability of Scientific Studies or whatever, something like that. And in any case, I don't want to limit our view of Simine to her advocacy of what she might call open science practices. In fact, Simine is currently an associate professor of psychology at the University of California at Davis, where she directs the Personality and Self Knowledge Laboratory. A lab dedicated to the study, not only of the stability of our personality traits, but also of how we know ourselves, and whether we know ourselves. Along the way, Simine somehow finds time for all this other stuff to like, for instance, she co founded the Society for the Improvement of Psychological Science, which goes by SIPS. That's in case you were wondering. She serves as editor in chief of the journal Social, Psychological and Personality Science, and senior editor at Collabora. And she co hosts her own podcast, which is called the Black Goat, which sounds a little ominous to me, but I don't think it's ominous. And she even writes the occasional op ed for places like Slate Magazine, as well as their own blog posts. Point is, she's busy. This is a busy person. And if all that wasn't enough, she's also been at the center of the national conversations that scientists have been having about everything from open science practices, like replication, transparency, data sharing, hypothesis, preregistration, all of that stuff. But also, and I think importantly, about the process of criticizing each other's work. About, you know, when, when criticism crosses over into harassment and bullying, and why that line is sort of both super important and a lot harder to identify than many of us might think. Now I have my own thoughts about all this, which I won't get into now, but I do want to say that I find Simine, in particular, to be a trustworthy conversation partner, as we go through all of these growing pains. I mean, I think it's telling in a good way, that Simine has been on the receiving end of pretty intense criticism from all sides of these difficult conversations we've been having. I think she tries really hard to be reasonable and even charitable towards people she disagrees with. And I mean, I don't know, I think in our age of sort of, shoot from the hip outrage, that can be a hard path to find, let alone walk. There are many things I love about Simine. But as you'll hear in a moment, at or near the top of the list of her agreeable traits is that she'll be the first to tell any of you that sometimes she is wrong. So friends, and foes, and relations, and other people for your edification and enjoyment. Here is my conversation with Simine Vazire.
Jim Coan
So you were just saying that you were here at the- So I just want to say again, that you're the first person to actually visit my home to do this. This is like your interview number 26 or so. And we were just talking about how you've been to UVA as a visiting scholar. Was right after you finished your work?
Simine Vazire
Yeah. So it was when I started a PhD and I had gotten the job at Wash U and I deferred it for a year. And I spent half the year here working with Tim Wilson and hanging out and eventually that led to the handbook that we edited together. We didn't start that while I was here.
Jim Coan
Wait a second. I didn't know but so what is this? What is this handbook?
Simine Vazire
The Handbook of Self Knowledge.
Jim Coan
Of self knowledge. Oh, that sounds scary. You know, too much self knowledg.
Simine Vazire
It wasn't really like a subfield at the time. And that probably still, would be an exaggeration to say that it's a subfield. But there were a lot of people working on things related to self knowledge and social psych and personality psych, and clinical and developmental and so on. And so we wanted to kind of bring all of that together into one place.
Jim Coan
Yeah, yeah.
Simine Vazire
But we didn't do that until after Yeah, when I was here, we mostly just hung out. I was dog sitting and house sitting Bobby Spellman's.
Jim Coan
Oh, yeah,
Simine Vazire
Dog. And so I would go to the dog park and see Tim Wilson there. We'd hang out all there.
Jim Coan
Me and Bobby would have had a party. We had a party here together where we brought all the dogs.
Simine Vazire
So she was away that semester. So I was staying in her apartment and ...
Jim Coan
Oh, yeah, I know exactly where that is. Yeah. Yeah. So John Height comes down still sometimes stays in there.
Simine Vazire
Yeah.
Jim Coan
I don't know why. I'm laughing. I just the topic of John Height apparently makes me laugh. That's just enough. That's all I need. Wow. So self knowledge. And that's really a big part of your domain of interest in personality, right?
Simine Vazire
Yeah. So it started out as more of a methodological question of like, How accurate are self reports? When should we trust self reports? And when shouldn't we?
Jim Coan
That's Tim's whole thing from decades, right?
Simine Vazire
Right. Right. And then, but I came at it from a totally different angle. I didn't come at it from like a cognitive like, What's going on people's heads angle, I came at it from a practical like- The first paper I ever wrote, as a grad student, we had self reports of our participants, but we also had pure reports from their friends. And for some incidental reason, we decided to use the peer reports in writing up the paper and that was our measure of personality. And the reviewers were very upset about this. And they said, Clearly, the best measure of personality is a person's self report. Otherwise, the whole field of personality seriously needs to rethink itself.
Jim Coan
Andyou thought, Yeah, the whole personality needs to rethink itself.
Simine Vazire
I thought this is an empirical question. Like, why would someone just state their...
Jim Coan
It is an empirical question.
Simine Vazire
So then I became obsessed with like thinking about- Yeah, I still don't know who that reviewer was, but I want to thank them because they like launched me in a totally different direction.
Jim Coan
That's the right attitude.
Simine Vazire
But so then I did a lot of like, more methodological work on like, for what constructs are self reports a good measure versus not and then I realized, it's actually a pretty deep theoretical question. Like, what do people know about themselves? What don't they? And that's how I got turned out into Tim's work and all the social cognition work on, you know, what the mental processes are that people use to form a self perception and...
Jim Coan
Oh, man, you know, we've started getting into this, because it's obvious, but you don't think about it until you think about it. Which is that self is an ongoing process that's unfolding in time. It's not a thing that you pull out of your brain and weigh and bounce like a basketball.
Simine Vazire
Well it's funny, my mom is a therapist, and I think sometimes it drives her a little bit nuts that I'm like a personality psychologist who reduces people to like five numbers. Because in her mind, like she doesn't believe that we even have one self, it's like, you have different parts of yourselves.
Jim Coan
Right.
Simine Vazire
So to reduce that, to answer even a personality questionnaire about your one self, I think is like, doesn't...
Jim Coan
It doesn't make any sense. Do you buy the whole...
Simine Vazire
Michelle?
Jim Coan
Michelle, Walter Michelle, the whole you now, I mean, I've thought about this, right? Because I'm, classically, if you take my numbers, I'm like, in above the 90th percentile and extraversion. I love to be around people.
Simine Vazire
Well it's going to be a fun conversations because I'm about 10th percentile.
Jim Coan
So I love- But here's the thing. I love being like- My platonic ideal of a social situation is the National Mall in Washington DC on the Fourth of July, right? Which is people everywhere.
Simine Vazire
Yeah.
Jim Coan
Fireworks.
Simine Vazire
Yeah.
Jim Coan
But if it's a party or something, especially if it's like a dance party, just suddenly I'm an introvert. Like I want to be off in a corner.
Simine Vazire
It's salient to you. Like the few situations where you're not extroverted are really salient to you. But so you're still like, I'm sure the vast majority if the aggregate across all the situations and over time, you would be our 90th percentile.
Jim Coan
Even with observational methods.
Simine Vazire
I think there's something to it. So I'm not sure I believe the fundamental attribution error is as robust as people say it is. But one thing that I think is really robust is that we can't see the forest for the trees when it comes to ourselves. We see these fluctuations and they feel like wild fluctuations. We don't see the stable average difference between us and other people. So like, my favorite example of that is my college boyfriend was your stereotypical computer science major, like pretty introverted.
Jim Coan
We'll leave that stereotype, what that means precisely. We'll leave that people's imagination.
Simine Vazire
Definitely very introverted. Like more introverted than me. Let's put it that way.
Jim Coan
Okay.
Simine Vazire
And I remember one year, we were still friends after we dated, and he read something in Time Magazine about ambiverts or whatever. Like-
Jim Coan
Yeah.
Simine Vazire
People who are sometimes introverted, sometimes extroverted, right? And he came to me and he said, Simine I get it, like, sometimes I'm introverted, and sometimes I'm extroverted. And I said, No, Nate, sometimes you're really, really introverted and sometimes you're introverted. But for him, that feels like the whole spectrum, right? But you don't see that like, your range is still a subset of the whole and even if you do occupy the whole range, you still have a part of that range where you're almost always in that kind of either high or low and that makes you different from other people. Like we really differ in our average states, even though we fluctuate around those. And I think actually Will Leeson, to me the prison situation debate is over, because Will Leeson resolved it.
Jim Coan
Will Leeson. Yeah, I haven't read that, shit.
Simine Vazire
So he kind of comes from the social cognitive approach to personality, but he reconciled it with traits by showing that people- If I measure your extraversion state over and over again. So maybe you use experience sampling or behavioral observations, yeah. But I repeatedly measure your state. Not your trade, but your state, like in the last hour, how extroverted were you? Or how much did you talk or whatever. And I can create a density distribution for each person. There are big mean differences. So your average state would be much higher than mine. But both of us would occupy the full spectrum, especially on extraversion partly because of situational demands, right?
Jim Coan
Yeah.
Simine Vazire
So if I'm going to teach, I can't be at the bottom of the spectrum on extraversion in that hour that I'm teaching.
Jim Coan
In that moment. Right. Right.
Simine Vazire
And even someone who's super extroverted, if you go to a lecture and sit in the audience, you can't be super extroverted in that context.
Jim Coan
Yes.
Simine Vazire
So because we have to occupy everyone that spectrum at some point, we sound we all vary a lot.
Jim Coan
So is part of the idea that certain situations have personality characteristics?
Simine Vazire
Yeah. So situations afford different, you know-
Jim Coan
Yes.
Simine Vazire
Levels of personality status, or some situations afford any level. Right? And then your personality, you see individual differences shine through for those what maybe a social psychologists would call a weak situation as strong situation.
Jim Coan
Yeah, this is great. This is real social psychology.
Simine Vazire
Yeah. So he opened up I mean, I recently co-edited a special issue. Yeah, yeah. So and, of course, an interaction too, right? So some- There's huge main effects of situations. But then there's the odd person who doesn't respond to the situation the same way that everyone else does. And that's interesting, too, right? So this person by situation interactions...
Jim Coan
Do you know about generalizability theory?
Simine Vazire
Yeah, vaguely.
Jim Coan
It's all about variance accounted for and reliability of one thing across other things, right? So you just don't worry about questions of significant differences or something. And one of the things that it has in it, it has different- It builds in different intraclass correlations. One thing you can do with generalizability theory is get an estimate for let's say, someone's extraversion score. In a reliability estimate, that's absolute, right? And you would expect that to be relatively low, right? But the other thing that I love that I really love is that it gives you a relative reliability score relative to other people in the dataset. And so what you very often see - what we saw - we published one study, one paper, where we showed that with EEG asymmetry, the absolute reliability is a pretty low across the situation. So it looks like you know, the situation is- But if you look at the relative reliability, so what it looks like is the distribution shifts up and down. But stays relatively stable.
Simine Vazire
Rank order stability.
Jim Coan
The rank order in instability is really interesting.
Simine Vazire
Yeah, I think it's really fascinating to look at. You can like flip a data set, transpose it, and then you see things that you didn't see when you're looking at it the other way. In personalities psych we like to do that a lot. So we have the nomothetic approach and then the ideographic. Then we flip it and we do profile. So
Jim Coan
Yes, yeah, yeah.
Simine Vazire
You get totally different results.
Jim Coan
Yeah that was, I think, when we were coming of age, in some ways, as grad students. That was, you know, it's sort of when we're also- We have like linear mixed modeling emerging and all of these kinds of ways to look at our datasets that really, really expanded the horizon.
Simine Vazire
Yeah, I mean, one thing that might take away from having these like datasets with, you know, repeated intensive within person measures and things like that. Sometimes the descriptive stats are the most fascinating. Like just looking at, across different constructs, what proportion of the variance is within person versus between person and how constructs differ on that?
Jim Coan
Yeah.
Simine Vazire
So some constructs, it's mostly within person. And then sometimes you'll see constructs where there's very little with inperson variance.
Jim Coan
Yeah.
Simine Vazire
And that in itself, you could write a whole paper just on that.
Jim Coan
I did.
Simine Vazire
Yeah, cool.
Jim Coan
The G Theory gave me the opportunity to just write in terms of variance components, instead of significant differences between means.
Simine Vazire
Yeah.
Jim Coan
And that was awesome.
Simine Vazire
Yeah.
Jim Coan
Can't do it very often.
Simine Vazire
Yeah.
Jim Coan
It's tough. Reviewers, you know, it's... I mean, I don't want to blame. I don't want to be one of those guys whose like, Oh, reviewers don't get how awesome I am. But I do wish that that things like G Theory...
Simine Vazire
I think that they's starting to be more flexibility in how people think about the papers don't have to fit exactly the old prototype anymore.
Jim Coan
Would you say that that's one of the... one of the side effects of I don't know what to call it. So here's what we're gonna have the movement can I call it that? Can I refer to you as an activist or is that too... is that overstating it?
Simine Vazire
I'm fine with that.
Jim Coan
Okay.
Simine Vazire
I don't know if it's accurate. It doesn't offend.
Jim Coan
I just want to say some- I want to be able to put a marker so so that we don't have to, you know, circumlocute everything. But do you think one of the the effects of the movement to increase the repeatability of our science is expanding our data analytic options?
Simine Vazire
I think that's one of the big debates, right? So I think some people perceive the movement as trying to restrict everybody into this one mode of like, We have to think of things a certain way and emphasize replicability over everything else, and so on. But I don't know. I mean, it's an empirical question. It'll be interesting to look at the meta science on whether the kinds of articles that are being published, whether there's a broader range or there's more variance in the types of articles being published post-movement, compared to pre. My subjective anecdotal impression is, yeah, I think we're seeing more papers with mixed results with no results, things like that. That wasn't possible before. More papers that don't use null hypothesis significance testing at all. Papers that are more explicitly exploratory or descriptive.
Jim Coan
Yeah.
Simine Vazire
So I think it is expanding the like kinds of papers people are willing to find. I think they're the papers that people had inside of them, but didn't think they could get published. Now they're willing to try and more reviewers and editors are willing to accept papers that don't fit the old mold.
Simine Vazire
Yeah.
Simine Vazire
Maybe that's too idealistic. But...
Jim Coan
Well, I mean, like I said, We'll have to wait and see. I mean, I've always been- Lee Secrest, my old methods mentor. So my minor was in methodology and measurement with an emphasis on measurement. And I really, I'm still sort of obsessed with reliability in these kinds of things. My own feeling is our measures is the root of- maybe the root of our problem.
Simine Vazire
Yeah.
Jim Coan
But I mean, that's a whole other cover- I don't want to hijack the conversation in that direction. But he used to call me a contrarian. It used to drive him crazy, because every time he would come out with some recommendation, mythologically it's true, I kind of suddenly wanted to do the opposite.
Simine Vazire
Yeah, I think that's-
Jim Coan
And I think there's some heuristic value in that in terms of moving things forward. Because you can't violate the principles.
Simine Vazire
Yeah. Yeah. I think a lot of academics, a lot of successful academics have that personality trait.
Jim Coan
It's weird. I don't know what that's about. So I want to do suddenly, I want to do like in on one studies.
Simine Vazire
Yeah. Yeah. No, that's awesome. I think expanding the range of things that we consider scientific contributions would be a great outcome.
Jim Coan
Yeah.
Simine Vazire
If that was one of the side effects of this movement.
Jim Coan
So speaking of the movement, I mean, you're like a superhero. Can I say that? Yeah I can say that. You've had tequila already. That you're one of the like key figures. Almost, I would say, I mean, I'm gonna say literally kind of a political figure in all of this. I mean, you're president of SIPS, the Society for the Improvement of Psychological Science. Right? I got it. So your're President right now?
Simine Vazire
Yeah. Chair of the executive committee, whatever.
Jim Coan
Chair of the executive committe.
Simine Vazire
So it rotates. The executive committee selects a new chair every year. So I'm the chair this year, and it'll be someone else next.
Jim Coan
Yeah, well, to me that's the president and you probably can command the armed forces as well. Whoever those people are, but I've got this list here. Like your're editor in chief of SPPS. Social Personality...
Simine Vazire
Social, Psychological, and Personality Science.
Jim Coan
Sorry, I can't...
Simine Vazire
You got the acronym right. Which is already like, you know, better than a lot of people.
Jim Coan
Really?
Simine Vazire
SPSS-
Jim Coan
I'm gonna take that. Yeah. Okay.
Simine Vazire
It's hard to keep them straight.
Jim Coan
You're doing- I can't remember exactly what your role is with collabora?
Simine Vazire
Senior Editor.
Jim Coan
Senior editor with Collabra. I don't know what- Okay. Your're current chair of the executive committee, president of SIPS. You've now got this podcast, the Black Goat. Which you've been doing now for what, a year?
Simine Vazire
No, just since February.
Jim Coan
Just since February. Geeze. You guys move fast. You guys are efficient and organized.
Simine Vazire
We're not perfectionist. Let's put it that way.
Jim Coan
Yeah, well, that's good. And you run a blog?
Simine Vazire
Once in a while.
Jim Coan
And you're pretty, at least you used to be, I don't know, you're pretty active on social media and things. I mean, I don't know because, I mean, this is one of the things maybe we'll touch on. About, I would say last March, or February even, I just bugged out. I pulled the chute and bugged out. I don't interact with any of those groups anymore. Because I couldn't. There was one too many nights of lost sleep worrying about whether tone was a thing.
Simine Vazire
Yeah I know.
Jim Coan
Whether I was completely crazy, or you know what.
Simine Vazire
Yeah, it's a really interesting thing about trying to figure out like, when should you try to have thicker skin versus when should you just disconnect and be like, I can't I don't have thick enough skin for this. I'm just gonna, like show myself out. Because sometimes, you know, I worry about like- So when I started a blog, I gave myself permission to not respond to comments if I don't want to. I don't have to engage with every critic or whatever. Otherwise, I would never have started the blog. Not because there are that many responses but just in my mind, I imagined what if this happens, this happens. And I was like, Okay, well, just because I put my blog up doesn't mean I have to engage with every response to it or whatever. But then that's an easy way to just avoid negative feedback or criticism so I don't want to be that person either.
Jim Coan
Thank God for that. I don't want criticism.
Simine Vazire
But finding that balance, right? Like, I want to leave myself open to criticism, and I want to listen to it. But then when I feel like the criticism is in bad faith, or I just or other shit is going on in my life, and I don't have the mental resources or whatever to deal with it, I want to give myself permission not to. But I don't know how that looks. If people think I'm running away, or you know? Like people don't- It's, you know, you don't always... It's hard. It's hard.
Jim Coan
Right.
Simine Vazire
People don't know... I don't know if people take into account the fact that there's a human being behind this. And so sometimes I'm not going to have thick enough skin to deal with it. Or sometimes there's other stuff going on in my life that's going to make it harder-
Jim Coan
Yeah.
Simine Vazire
To deal with it. But if I use that as an excuse not to engage at all, then I would just never engage. So in my mind, I gave myself permission to say, Okay, look, this might look bad and it might sometimes look like I'm a coward, but I'm going to let myself put some stuff out there and then walk away and let-
Jim Coan
Just let the world go crazy. Yeah.
Simine Vazire
And they do the same with the online, the Facebook forums and stuff. It's like, sometimes I'll chime in. And then sometimes I'll chime in, and then leave and not look at what happens afterwards. Because you know, I'm, you know, I need a little bit of space or whatever.
Jim Coan
You know, the main thing that I worry about- And you know, it's funny when you- we're talking about what this podcast is about. And if it was ever about anything in the early days, I sort of had this fantasy that I would get every guest on every side of some debate, to say something inappropriate. And just let the world deal with it. Right?
Simine Vazire
That's why we were so eager to give me a shot of tequila.
Jim Coan
Yeah, that's right. I want to see something happen. But I don't really think that's true. But I do feel like the outrage machine is a little too... is a little out of control.
Simine Vazire
Yeah.
Jim Coan
You know? The thing about being at a bar, and this is this is the canonical, you know, example. Although Sanjay, took me to task about this one time and I'll tell you about that in a sec. But you know, you go to the conference, you sit at the bar.
Simine Vazire
You say what you really think.
Jim Coan
You say what you really think. And sometimes you're kind of a dick. Or you know, you just get it wrong, or you use an inappropriate analogy, or whatever. Because you're just- Because the thing is, when you're on the fly, you're sort of groping for the right thing, and you don't really know what it is yet.
Simine Vazire
Yeah.
Jim Coan
And so you're gonna do some wrong things. Much like you'll have a paper that has incorrect finding or something.
Simine Vazire
Yeah. No, I've stepped in a few times.
Jim Coan
But the hammer comes down.
Simine Vazire
Yeah. I think you know, I don't know, this was maybe too much faith in my fellow human beings. But I think that if you're willing to say, Oh, yeah, sorry, my bad. I shouldn't have gone there.
Jim Coan
Yeah.
Simine Vazire
I've done that. One time I made a terrible analogy and people called me on it.
Jim Coan
I did that yesterday.
Simine Vazire
That was insensitive.
Jim Coan
Yeah. Yeah.
Simine Vazire
So what are you gonna do? I hope that you know, that people are understanding if you react non defensively. I think that's the key.
Jim Coan
Yeah.
Simine Vazire
And so far, from what I've seen, I think people will back off if you do.
Jim Coan
You know, John Gottman has this great view of fighting that's evolved over the years. Because, you know, I did my early work with him. For a long time, I ran his labs up in Seattle as a lab coordinator, and I've gotten to know him pretty well. You know, in the early days, it was all about sort of identifying how people fight and trying to get them to avoid certain behaviors. And it didn't work out so well. You know, in terms of- It worked out beautifully into the predicting outcomes, right? Because you you can flag the people that are doomed.
Simine Vazire
Yeah.
Jim Coan
But in terms of intervention-
Simine Vazire
It's hard to change.
Jim Coan
He's really come to believe that shitty behavior is on the menu, whether you like it or not. So the key thing is sort of, as you're saying, is how you recover from it. Right?
Simine Vazire
I'm still working on that.
Jim Coan
Aren't all of us. All of us are working on that. Because it's really hard. You get your feelings hurt or whatever. And how do you deal with it?
Simine Vazire
Yeah.
Jim Coan
Or you just get your backup about what? You know- God knows what it is. Only the point is, there's always going to be something.
Simine Vazire
Right. right.
Jim Coan
Right?
Simine Vazire
Yeah, I mean, I think that also face to face interaction is really important for that. Because it can be so easy for things to escalate online or from a distance, but then you actually meet the person in person. And I mean, my favorite example of that recently was Daniel Watkins and I are on opposite sides of this point o o five output thing.
Jim Coan
Yes, I read that. I'm a lurker in all of these debates.
Simine Vazire
And then we- Like the day after all that hit and we, you know, both were pretty clear about our positions online. We took a nine hour train ride together to a conference, and it was great. And I don't think in that particular case that it would have gotten ugly if we hadn't had that face to face time. But the face to face time... It could have yeah.
Jim Coan
He's not- I'm going to go ahead and say this, he's going to listen to someday. Maybe. That's a terrible assumption. But he's not great online. I would give him that feedback. If he were right here, I'd say that he stumbles in ways online that bum me out because it's unnecessary, given what he's really like.
Simine Vazire
Yeah. In my experience, he also exemplifies this. Like being able to say, Yeah, I shouldn't have done that. I shouldn't have said that. I shouldn't have gone there. Which you know, but then you could say, Okay, but after a few trials and errors, you should-
Jim Coan
Cut it out.
Simine Vazire
Change. But I think most people are way more interesting to talk to you face to face than on Twitter. And so I think it's important if possible. And this is, you know, there's a lot of talk about, like, how much inefficiency there is in like, you know, with climate change, and everything. Like flying to conferences is not the way to go in the future as online. And I have to admit, I'm ambivalent about that. Because I think actually being face to face in the same room has a lot of value. And I know that that means that's a huge burden on people without resources, and it's a burden on the environment and so on. I just-
Jim Coan
Yeah.
Simine Vazire
I don't know what the solution is. But we need face to face time too.
Jim Coan
Well, I mean, frankly, that's a big motivator for what we're doing right now. You know, Sanjay, you know- There was some Facebook, interminable Facebook exchange. But this is also an illustration of how they can be helpful. You know, I was reminiscing, I was in the warm glow of these wonderful times I've been at conferences talking with senior colleagues about times we couldn't replicate a thing and so it really helped me, you know? And he's like, Well, you know, yipee for you, but a lot of people didn't have access to that conversation. So I was like, Huh, that's, yeah, shit.
Simine Vazire
Yeah. Yeah. I mean, it's interesting. I feel like I'm being like apologist for face to face interactions and so on. But at the same time, I think a lot of people shit on social media. But it has so many positives. Like, that's a great example of like, if you walked away from that conversation, and then tweeted, like, Hey, guys, a lot of people can't replicate this effect. That would have a lot of positive value for people. And especially for people who are generally at a disadvantage.
Jim Coan
Yes.
Simine Vazire
So I love social media.
Jim Coan
The thread there, you're talking about somebody's research.
Simine Vazire
No I know. Yeah. So I have a complicated relationship with criticizing specific bodies of literature. I tend not to do it myself and then I feel guilty that I benefit from the people who do call it out. Then I don't have to do it. And you know. So I think we need to find a way to be able to express skepticism about a line of research.
Jim Coan
Yeah.
Simine Vazire
Without it... Yeah, I think it's complicated. We haven't figured that out.
Jim Coan
Sometimes I wonder if my view of this sort of at the bird's eye level is a little bit different from a lot of my friends and colleagues in social psych. Because I've been doing psychophysiology for a long time. In the Society for Psychophysiological Research SPR, I don't know how much you know about it.
Simine Vazire
Not much.
Jim Coan
But it's been around for more than 50 years now. They were the first psycho phys conference and journal. And it's been deeply methodological from the very beginning. And a lot of these growing pains, I think, SPR as a society went through a long time ago.
Simine Vazire
It's interesting, because I think a lot of personality psychologists would say the same thing about their area that-
Jim Coan
Interesting.
Simine Vazire
Like, our conferences are like... Well, they've now gotten down to like, maybe 25% is factor analysis.
Jim Coan
Right.
Simine Vazire
So it's always been very measurement oriented. And like, yeah, a methodologically oriented field.
Jim Coan
Yeah.
Simine Vazire
And so I think that-
Jim Coan
I say that in part, because I've seen so much open criticism of each other's work at these conferences. Is that the sort of thing that you see in personality psych all these years?
Simine Vazire
Yeah. Certainly, there aren't sacred cows, I would say in personality psych. There's nobody that you can't criticize. There's nobody who's like such a hero, that- Which is something I feel sometimes in social psych. And I know, it's a little dangerous as a personality psychologists to criticize social psych. I love social psych. I think there's many things that social psych does better than personality psych. So I don't want to sound like I think that personality doesn't have any flaws. But one of the things I do like about personality psych is that I can disagree with the biggest names in the field, or my mentors, or whoever, and it's all in good fun and people kind of relish the debate.
Jim Coan
Yeah.
Simine Vazire
And there aren't- It doesn't become personal. So that, you know, there's camps of like, five factors versus six factors.
Jim Coan
Yeah.
Simine Vazire
Traits versus non traits or-
Jim Coan
Or single factor even.
Simine Vazire
Yeah. And there's- The friendships don't always fall along the same lines as the intellectual positions. So that makes things a lot easier.
Jim Coan
Yeah, that's nice.
Simine Vazire
Yeah.
Jim Coan
That's sort of idyllic in a way.
Simine Vazire
Yeah.
Jim Coan
I remember coming out with a book chapter, where I severely criticized the whole research tradition of frontal EEG asymmetry. Because I've been doing it as a graduate student, and it was a pain in the ass. Like, I mean, so much of the time, I couldn't make anything happen. And I couldn't figure out what was going on. I was doing everything right. And then sometimes, poof! There it is. I can see it. And so it was really a mystery and I was very frustrated and I felt like my career was doomed.
Simine Vazire
Yeah.
Jim Coan
So I wrote this thing. And Richie Davidson, who's like, invented it, right? Invited me to be his postdoc.
Simine Vazire
That's awesome. That's great. Yeah, that's a sign of a really healthy field, I think.
Jim Coan
Yeah, I think so too. So okay. So, can I just bring up some of the, the prescriptive advice of the movement? Can we go through some of this stuff? Because I want you to hear my confessions.
Simine Vazire
Okay.
Jim Coan
And I'm going to the top. So one of the big debates that's come up is direct versus what's been called sort of conceptual replication. How would you characterize that debate? Could you characterize that for me, briefly?
Simine Vazire
So like in one or two sentences? I would say the various positions are one that conceptual replications are enough because they're better than direct applications because you have the chance of learning something new and you add to the generalizability of the phenomenon. And the other view is that no, we need direct applications because when a conceptual application fails, there's a lot of ambiguity. And so, if we only do conceptual replications, the original finding can never be falsified if it turns out to be a false positive, because there's always alternative explanations for why the conceptual or application failed that are very plausible.
Jim Coan
Too many researcher degrees of freedom.
Simine Vazire
Well, and not even-
Jim Coan
Pulling the posthoc explanation.
Simine Vazire
Yeah, and too many legitimate possible reasons.
Jim Coan
Yeah, yeah.
Simine Vazire
Right. So if you extend it, you don't know if it failed because of the extension, or because the core phenomenon isn't reliable.
Jim Coan
Yeah.
Simine Vazire
But, you know, the easy solution is to build an indirect replication and an extension, and then- Kind of like the same idea behind a manipulation check. Right? If it fails, you want to know, was it because I didn't manipulate the independent variable? Which is still important to know. That still says something about the original finding.
Jim Coan
Yeah.
Simine Vazire
If the manipulation check fails, that you should wonder why it worked in the other one? Or did it really work in the other one?
Jim Coan
Right.
Simine Vazire
And the same goes for direct and conceptual replication. That if it's the conceptual replication fails, you should want to know, is it because the the direct part of it wasn't robust? Or is it because extension part of it wasn't?
Jim Coan
It's funny. Two responses to that I've had over the years that has come up as a conversation. One is that in practice, I've been doing that for a long time and I think a lot of people have. Which is funny, because theoretically, I've always said, Don't bother with the direct replication. So, you know, I'm saying one thing with my mouth and doing another thing with my feet. And I've been doing that for like- For example, you know, when the... And again, I am not intending to pick on anybody. But you know, the the glucose dip, following the Baumeister-
Simine Vazire
Ego depletion.
Jim Coan
Ego depletion finding. When that first came out, I was thunderstruck. I was like, Oh, my God. It's like, we have a currency to deal with for cognition and that's just such an advance. So I wanted to do some of that. And I wanted to manipulate it in different ways using social presence and things like that. But I couldn't. You know, I'm gonna say it, I couldn't replicate it. I just couldn't. And I tried a lot of times. And I contacted people, and I tried to work it out. Because I was absolutely- I was seriously distressed by that. But I intuited that I wanted to just see if I could do it first before it started moving it around.
Simine Vazire
I think you're probably right, that a lot of people were doing that. But it wasn't getting out.
Jim Coan
It wasn't getting out into the world. Unless we were having conversations like this at the bar at conferences.
Simine Vazire
I think even when it was, it wasn't leading people to change their minds about the original phenomenon a lot of times. It was still like, people were dodging the conclusion that maybe the phenomenon was a false positive or wasn't robust. And again, like, I think we should be conservative and jumping to that conclusio. It should take a lot for us to reverse our view on something. Like thinking about it from a Bayesian perspective, like if we have a strong prior if we have strong evidence to begin with, then one field replication shouldn't be enough to reverse that. But it should be possible. And I think the situation we were in before was that, there wasn't a mechanism for the consensus in the field to change. A lab could change their mind and say, Yeah, I'm not going to try to build on that, because I've tried to replicate and it doesn't work. And that might spread to a few other labs. But I don't know of cases where it spread to the field and the field was like, Oh, yeah, no, let's not go down that road. That turns out not to be fruitful.
Jim Coan
Yeah.
Simine Vazire
So I think what many of us who want to promote more direct replications want is a mechanism. We say science is self correcting. But it's hard to think of examples where we really walked all the way back on a finding
Jim Coan
On a finding. Yeah.
Simine Vazire
And that should happen. In a healthy science that should happen. And we're not seeing it happening very much until recently.
Jim Coan
Well, yeah. Part of that is because we have weak theories, but then we have weak theories, because we have bad measurement and, you know, blah, blah, blah. It goes on and on and on. It's a mess. But yeah. But so in my training, in my measurement training with Lee Secrest, he drummed it in my head that direct replications were useless. And here's why. So we had to read David Liken's old piece from like, 1968. You know, where he lays out different kinds of replication. It's a great paper and it's really illuminating because here's what he said. Licken was a grumpy old guy, right? He's the grumpy old psychophysiologist.
Simine Vazire
I think I read a book he wrote.
Jim Coan
Maybe Tremor in the Blood, about lie detection, which is bullshit. But the argument that he made was he didn't use conceptual replication he used, constructive was his word, constructive replication. And what he argued was that every finding, if it's a legitimate science, it contributes to our understanding of a theory. The theory has a set of propositions. It gets very Poparian right, in here at some point. And he goes, what needs to be replicated is the proposition, right? And so what you don't want to do is depend on what some other jerk did. Not jerk. Some other very fine, wonderful person did in their study, because they might have done it a bad way. What you want to do is test the proposition. If the proposition- If you falsify the proposition in a way that they didn't-
Simine Vazire
That would only work if there was consensus that you did it better than them.
Jim Coan
I know.
Simine Vazire
So I agree. I would rather instead of a direct replication, you did a an improvement on the original study. The problem is, the original authors or their friends, or people who liked that original finding, are not going to agree that you did an improvement. They're gonna say what you changed, if you fail to get the effect, what you changed made it worse. So if we could come to a consensus on what's an improvement, and everybody agrees, Yes, this is an even better test of the theory. Then I would be all for changing the-
Jim Coan
Right.
Simine Vazire
The design from the original to the replication.
Jim Coan
I think Liken would have argued that what we need to be arguing about is what the best test of the theory is.
Simine Vazire
But then we just devolve into he said she said, and we never correct the record when there's a false positive.
Jim Coan
Maybe the place we're Liken and the sort of broader consensus of the movement agree is that the lowest rung, the smallest hurdle-
Simine Vazire
Yeah.
Jim Coan
Is the direct replication.
Simine Vazire
I think that really the disagreement is not about the value of direct replication. I think we all agree that it's a pretty, pretty low bar. It's a matter of cynicism. I think those of us who are pushing for more direct replication think that we're worried that we won't even pass that low bar. So we think before we skip to extensions and nuance about the theory and boundary conditions and so on, let's make sure that these facts that we're incorporating into the theory are facts. And they'r not false positives.
Jim Coan
Yeah although you can make the argument I think that Liken and Secrest are even more cynical.
Simine Vazire
Yeah.
Jim Coan
They're like, If you can't take the theory and make it happen with another clever design, then the theory is bullshit.
Simine Vazire
I agree. But the field doesn't share that. consesus.
Jim Coan
Right. Right.
Simine Vazire
So but I think we need to move towards better standards of what's high quality evidence. So that if I don't do a direct replication, if I test the same theory, but in a better way and it fails, then it's harder to dodge the conclusion that. Okay. well, that prediction was wrong.
Jim Coan
Right.
Simine Vazire
But right now, there's not enough consensus on what makes one instantiation of the theory better than another. And so then people can choose whichever finding they want to believe.
Jim Coan
It's, yeah. It's- This is part of why I don't know whether I'm, like, more or less grumpy about the state of psychology sometimes when I listen to you.
Simine Vazire
Right. I mean, cynicism is a complicated thing. I think in some ways, I'm super cynical. And in other ways, I'm super idealistic.
Jim Coan
Yeah, you have- Well, I think that the history of movements of any kind suggests that you have to be idealistic in order to participate. And you're like the movement. You're one of the-
Jim Coan
I just want to go on record that I don't agree with a lot of the characterizations of me personally, that you're making, but I'll let that slide.
Jim Coan
That's too bad. I'm just kidding. No, you're right. Of course. Of course you're right. I'm sort of joking when I say that you're the movement. But to me, you're sort of one of the very nice- the sort of the bright lights of the movement. And I mean that in a good way.
Simine Vazire
I just want to say that one of the- Like, maybe the biggest lesson I've learned from the movement is that people who get put on pedestals, that's not a place you want to be. So I get uncomfortable when people describe me, like, yesterday, someone called me a big wig. And I know they mean it in a nice way. But to me, that's like my worst nightmare. Like I don't want to be in that position. Because I'm so far from perfect and when people realize that I don't want that to be a big fall, right? I want to be like, pretty close to the bottom so that when people see all my flaws and see that I'm not perfect, I'm not falling from a great height.
Jim Coan
I bet that also you want your disposed to affiliate.
Simine Vazire
Yeah, I mean, one of the things that was interesting, I'm curious what you think about this, but one of the reasons I like doing a podcast and we talked about this is that, I like deflected attention. So I like in this role, actually in SIPS. Like, technically, I'm the chair of the executive committee, but Brian Nozek, and Alexa are kind of the MCs of the conference, because I am not good at like having the spotlight directly on me, but I won't deny that I like to have an influence and so on. But I like that to be a little bit more indirect or deflected. So the nice thing about the podcast is that you can record it and imagine that nobody is going to listen.
Jim Coan
Right.
Simine Vazire
That's what I have to do when I'm recording a podcast.
Jim Coan
Okay, well, that's interesting. And maybe this is part of why when I saw you at APS last May, I asked you about editing SPPS. And you were like, I love it. And I was like, You're crazy.
Simine Vazire
I love editing.
Simine Vazire
I can't stand it. Yeah, I can't see. I can't take the pressure. Yeah, I like being up on stage man. I can be upstate on stage all day long. I don't care. I like interacting with the audience.
Simine Vazire
It's good that different people like different parts of the job because otherwise nobody would want to do certain things.
Jim Coan
Man, I just feel so responsible for weighty decisions and that's where I crack.
Simine Vazire
Yeah.
Jim Coan
Because my other high score on personalities neuroticism.
Simine Vazire
Yeah. I think I'm pretty low on most facets of neuroticism. And I think that does help for editing. I mean, I don't want to sound like I don't care, I do. It does weigh on me that I'm making decisions that affect people's livelihoods and lives.
Jim Coan
Yeah.
Simine Vazire
But I think that I can... That doesn't stop me from being able to make a decision. Knowing you know, I've been on the receiving end of bad decisions and mistakes and so on. Knowing that I will sometimes make some mistakes and I don't take that lightly. But yeah, I can understand how that's paralyzing for some people. For some reason, that doesn't paralyze me, but it is on my mind.
Jim Coan
Yeah. All right, well, so what about preregistration? What is this? Can I- Here's my confession part, really. Well, my first confession is, I'm still not 100% convinced about direct replication, but I liked that we found that low bar. We agree that it's a low bar, lowest bar. But preregistration. Haven't done it.
Simine Vazire
Yeah.
Jim Coan
So what does it entail?
Simine Vazire
So I haven't done it either, except for like, for small side projects that I'm not the main person on. I have been involved in projects that have preregistration.
Jim Coan
So you suffered through the-
Simine Vazire
So I haven't collected new data in like five years.
Jim Coan
Really?
Simine Vazire
Yeah, because our data, our longitudinal multi method, measurement, burst designs with behavioral observation and self reports and peer reports and life narrative interviews. So we're still coding all of that from our last study. And we will be for a while.
Jim Coan
You've got longitudinal data that makes pre registration tricky.
Simine Vazire
Yeah and we don't have hypotheses, we don't have predictions.
Jim Coan
You don't have predictions?
Simine Vazire
If I could go back in time, I would pre register my last study as an exploratory study and just say, This is what we're gonna measure. We have some research questions, open ended research questions. We have a lot that we haven't thought of yet that we will use the data to answer anything new ideas that come up. But I think that's still worth preregistering because it ties our hands it makes it so that we can't later claim that we predicted something. So the-
Jim Coan
A lot to unpack there.
Simine Vazire
Yeah.
Jim Coan
So what is what is preregistration anyway.
Simine Vazire
So preregistration is if you want to later be able to get credit for having had a specific prediction, then you should document it, right? Like we know about motivated reasoning. We know that there's retrospective biases, we know that there's motivated biases, people confirmation bias, etc. So if you later want to write up a paper and say, We expected this before we ever saw the data and then we tested it. We had one key test, which is really what the conditions that need to be met if you're using p values in the Neyman Pearson way. So if you want to be able to interpret your results with as much confidence as you would if you had a key test of a key hypothesis that you made ahead of time, then show us that. You know, it's hard to ask people to believe that to trust you that plan all along, because of motivated reasoning and retrospective bias. Because we're human. It's so easy to think that we had a prediction all along when we didn't. When you play around with the data, you see something interesting and then you can reconstruct, as if that was the goal of the study along. I've seen it happen with collaborators, where we have a paper where my collaborator and I disagree about whether we predicted the finding ahead of time or not. Like we both actually- We have that vivid memories, where we were sitting. Yeah. But that's what so much of the social psych literature is about.
Jim Coan
So for example, you know, I have these studies with hand holding, right? And you know, I put people into threat of shock and I have them either hold hands with a loved one or not. And the prediction was that they would be less distressed when they're holding hands.
Simine Vazire
But let's say you test that, and it doesn't quite come out and then you think, Oh, yeah, but there might be gender differences. Or it might matter how close they're close one is. And so you add that as a moderator or whatever.
Jim Coan
Yeah.
Simine Vazire
And then, you know, it's hard to remember how many different things you tried if the first one didn't work out, and to adjust. First of all, we're terrible in our intuitions about probability. So how do we adjust our confidence in the effect depending on how many ways we had to try it? How do we remember how many things we tried before we landed on the one that stuck?
Jim Coan
Yeah.
Simine Vazire
And how do we know whether we would have thought a priori that was an obvious moderator or not. I don't trust our retrospective. We're really good at storytelling and coming up with a good reason why that should be a moderator. Why wouldn't have worked without controlling for this variable and so on.
Jim Coan
I'm gonna have my mom call you. She needs to hear about this. Because when I when I first came out with that- She's gonna hate that I'm telling the story publicly. But when that first hand holding study came out, you know, it got press. And so I called my mom and was like, Hey, I'm in the paper. And she's like, what? And then she's reading it and reading it and reading. She's like, So you found that when you're holding hands with someone, they're less distressed when you put them under threat or shock? I was like, Yeah. She goes like this. You got that in the New York Times? Why? Why didn't you call me. I could have saved so much money! This was-
Simine Vazire
Yeah, I mean, I think that... I think your point, the cases where it's just the main effect, and there's really only one way to test it and so on. Then I'm less than worried about researcher degress of freedom.
Jim Coan
Yeah.
Simine Vazire
But many effects that we read about in the literature are more complicated, like interaction effects or mediations or mediated motivations and so on. And then-
Jim Coan
Intuitively, I never trust them as much.
Simine Vazire
Yeah. And I think the reason is that we know there probably were a lot of ways to test that. There probably were a lot of different variations on the hypothesis that all are theoretically plausible and easy to come up with the rationalization for. And some of them must be true. So it's not the case that just because it was slightly post hoc, that it's not true. But our confidence should be less than if that was exactly the prediction that the researcher made ahead of time.
Jim Coan
And so, does it get tricky, though? I mean, it's so... A pre registration is- I don't even know exactly what it is in terms of what the documentation looks like.
Simine Vazire
Yeah, so-
Jim Coan
But how does it- go ahead.
Simine Vazire
There's different degrees, right? So you can pre register just the methods and not say that- not have any predictions or analysis plan. And in that case, then your analyses will be necessarily will be exploratory and you'll have to interpret them with more uncertainty, more caution. Or you can pre register not just your design and procedures and materials, but also your key research question or hypotheses and how you're going to test them. You could go as far as to write the code that you're going to run when the data are in. And that doesn't mean that's all you can do. But that's the part that you can then claim as confirmatory and you could have more confidence in it. And then anything else you do you just have less confidence in than the stuff that you preregistered.
Jim Coan
So is the... Is there murky territory here? I mean, so I'm thinking about when I've written a bunch of grants. And grant always make you state hypotheses and data analysis plans and all that stuff. Does that count?
Simine Vazire
We could... We could decide that that counts. I think there's a lot to think about. I would- Yeah, I think I would be okay with that. If people wanted to count that as a pre registration. Yeah, I'd have to think more about it. But I think it's not too far from what we mean by pre registration.
Jim Coan
I'm fishing for some help here because I might be able to claim some pre registrations.
Simine Vazire
Yeah, I don't know. I think our field needs to come to consensus about that. And I don't know enough about what level of detail goes into those kinds of things. I know, my own case, but my research is pretty far from the mainstream in terms of design, and so on. So I don't know what's typical. I don't know how detailed those plans are. I don't know how much, you know, often there's a pretty big time lag between when you write the grant proposal and when you actually do the study. So it's possible that the design changes quite a bit. The predictions changed quite a bit, etcetera. But if they don't change, I don't see... I'd have to think more about it. But I think it could count as a kind of... Maybe, depending on- And like I said, pre rregistration is kind of on a spectrum. So depending on the level of detail and the level of precision in the description, it could be a more constraining pre registration, which case you can make really strong conclusion.
Jim Coan
Yeah, yeah.
Simine Vazire
Or more and more fluid pre registration, which case it's a little bit better than no preregistration, but maybe not as far as much confidence as a detailed one.
Jim Coan
Yeah, I mean, I've posted sort of half jokingly on Facebook, I was like, Hey, what if I put in my paper that I did just the exact analysis plan that I put in my grant? Because I've never seen anyone do that before.
Simine Vazire
Even the exact analysis plan that you used in your past paper. I think that anything actually constrains researcher degrees of freedom, it helps. And I don't know where the line is between something that helps and constraines it a little bit versus a full blown pre registration, but I think-
Jim Coan
And it's probably gonna vary from domain to domain.
Simine Vazire
Yeah. I mean, and it's hard because sometimes in some domains, it is legitimate for the decision rules about analysis to change from study to study. Sometimes you exclude outliers based on three standard deviations sometimes it's two and a half. But it could be a principled change.
Jim Coan
Oh, absolutely.
Simine Vazire
And so that's where predistortion comes in handy is then you can say, No, no, I know it looks weird, because in study one we excluded them based on three standard deviations, study two is two and a half. But we preregistered it. It wasn't a post hoc decision. It wasn't data dependent.
Jim Coan
It's tricky though. I mean, I totally- First of all, I totally see, conceptually, the advantage. Part of my resistance is that I just don't want to do another goddamn thing.
Simine Vazire
Yeah.
Jim Coan
I don't want to do another thing.
Simine Vazire
Liz Dunn gives talk at SPSP and she made a YouTube video of it. So she had, I think, some reservations of pre registration. I think probably similar ones.
Jim Coan
Yeah, because it's a pain in the ass. I mean, conceptually, it's totally onboard.
Simine Vazire
Yeah. So I think it's less a pain in the ass than you might think. And then you save time later on. And so again, like, I'm probably not the best person to talk about it, because I haven't done it. What we've done is, because we use existing data that we collected six years ago and we're still coding and so on. We can't do a data independent analysis plan because we know too much about our data. So we still register our plan analyses, but we don't claim that they're pre registered. We don't claim that it's independent of the data. But even that- It does, it's hard. We have to like pin ourselves down. So there were like six different ways we could test our research question. And to have to decide ahead of time which one do we think is actually the best test. Like if we had to pick one, which is like the best test and the other one which one are we committing to believing more?
Jim Coan
Yeah.
Simine Vazire
We're gonna like, kind of tie ourselves to the mast a little bit and not give ourselves complete wiggle room. We still have some wiggle room because we still say, Well, okay, we're gonna- This is our key test. But we also think it's worth trying at this for other ways.
Jim Coan
Yeah.
Simine Vazire
But if the key one doesn't come out, then we're at least blocking ourselves from making really strong claims.
Jim Coan
Interesting.
Simine Vazire
And it slows us down at that point in the process, but it speeds it up later on the interpretation side. We don't spend hours going around around and debating about, Well, no, I did. I would have predicted. I totally would have guessed that this would work better than that, and so on. Because we had to pin ourselves down earlier on.
Jim Coan
Well, this is really interesting in light of something that you said that I didn't expect a little while ago. And I just want to bring it up. We're running out of time. So I don't want to make you late. But you said that you have this longitudinal study and that you don't have any hypotheses. And can you talk more about that? I mean, I didn't know that. Let me just, first of all, I'm not pouncing or anything.
Simine Vazire
I don't feel at all.
Jim Coan
I just, I guess, philosophically, where I come from intellectually is that that's not possible.
Simine Vazire
Right. So I did have a research question. And it's in my NSF grant proposal. And it's that we want to look at whether people have self knowledge, not just of their average trait level, but what we were talking about before with Will Leeson in the within person fluctuations. Do people know how much they fluctuate, and when they're particularly high or low? Both on a short term scale. So we have like, one or two weeks of intensive within person data, self report on ESM, and observer ratings using the ear. So for a week, every year, we recorded people intensively and we got their self report and their actual behavior.
Jim Coan
Cool.
Simine Vazire
Then we did it again, a year later. And a year later, although we had a ton of attrition.
Jim Coan
That sucks. Sorry.
Simine Vazire
And we have self- Yeah, we did not know how to do longitudinal.
Jim Coan
Yeah.
Simine Vazire
So the kind of broad research question driving the study design was, Are people accurate about the state personality, how they're fluctuating from one situation to another one day to the next? And also one year to the next. Do they know how the personality is shifted over time? What about pure reports? Are those more accurate, less accurate, etcetera? So it's kind of open ended questions. But when you're going to design a study like that, you're going to also throw in everything but the kitchen sink, right? Like, so we had a collaborator who's interested in conscientiousness, and school achievement, and so on. So we have measures of that. And we have- We threw in measures of emotion regulation,
Jim Coan
But does your collaborator who threw in the conscientiousness measures have hypothe- Why did they choose the conscientiousness measure?
Simine Vazire
I think that a lot of like, personality researchers have these kind of open ended questions of how does this change over time? Or what predicts changes in conscientiousness?
Jim Coan
So they're just interested in conscientious? But they don't know what's going to happen with conscientiosness?
Simine Vazire
Yeah, I don't want to speak for him. I don't know. Maybe he had more specific-
Jim Coan
Sorry, sorry. Yeah, I don't want to put anyone in the spotlight.
Simine Vazire
No, but I think it's very common, at least in personality research to have a question- Be interested in a construct or the relationship between two constructs, or how construct develops over time, and be open to any, you know- Start with a very kind of bottom up approach of like, what correlates with this? And then dig deeper into it.
Jim Coan
So do you have an expectation for what sources of variation in personality measurement are going to be primary or secondary? Or how they're going to manifest and things like that?
Simine Vazire
What do you mean?
Jim Coan
Well, I mean, I'm trying to think about your- So you've got, you know, self report, you've got other report, you've got situational measures, you've got all these kinds of things. And you've got- You're very impressed with the idea of situations as having a sort of drawing for different personality traits. So what do you expect to happen? I mean, maybe not specifically, but you do have expectations? And would not count as hypotheses?
Simine Vazire
My answer to those questions is super unsatisfying to people. Like people often want to pin a person down and say, Well, you're for self reports, or against self reports, and I'm like completely in the middle. Like, I think that, in theory, I think there's a lot of blind spots in people's self views. And I think there's a lot they don't know about themselves. And if we could measure behavior really, really well, we would find things that people don't know about themselves and predict their future behavior and their outcomes better with behavior than with their self use sometimes.
Jim Coan
Yeah.
Simine Vazire
But the measurement of behavior is so much messier and so much harder than the measurement of self reports. We have such good self report measures. They're reliable, they've been validated, etc. So they end up predicting things better than the behavior measures. So if you... If I had to bet money on what's going to be most powerful, most predictive, etc, most valid and reliable, it's the self reports, and maybe the peer reports would be up there with them. But if- What do I really believe is who a person really is? I would probably go with behavior if I could get a direct pipeline and measure it really well, but it's so hard to do.
Jim Coan
Just follow them around with multimodal measurement system.
Simine Vazire
Right.
Jim Coan
But yeah.
Simine Vazire
And we have. I mean, we have 45 coders in our lab coding behavior. We triple coded everything and that's not reliable enough. So we're adding fourth, fifth and sixth coders and our reliabilities are still going to be below the standards for reliability.
Jim Coan
Like what? Like point four?
Simine Vazire
Yeah.
Jim Coan
Five?
Simine Vazire
Point 4, point 5 with three coders.
Jim Coan
Which they sometimes called acceptable.
Simine Vazire
Yeah. Yeah, I mean, into a personalized psychologist who uses questionnaire measures. That's disgusting. Like that makes me want to puke.
Jim Coan
Oh, yeah, that's not acceptable to me.
Simine Vazire
Yeah. So I don't know. But it's that or nothing with real world behavior.
Jim Coan
Well, I've been coding behavior for frickin almost God, I started in 91.
Simine Vazire
Yeah.
Jim Coan
I've been coding behavior for a long time I feel your pain.
Simine Vazire
Yeah.
Jim Coan
That part sucks. Reliabilities. One of the things that I was up against when a long time ago with frontal EEG asymmetry was I had this suspicion that one of the problems was the measurement conditions. That's why it wasn't replicating. This turned out to be right. That measuring at rest, was like an uncontrolled experimental condition. Like there wasn't enough situation in there to draw out the personality. So it was just like, people were just boop. All over the place.
Simine Vazire
Now I can ignore my minus three standard deviation score.
Jim Coan
Maybe so, maybe so. It could have been- Well, could have been the situation. Could have been your friend's fault. But so the flip side of that was that, you know, if we impose some kind of emotional situation, we'll draw out the response to that situation, but we'll also within that situation, we'll get people's trait, because we'll get this trait capability. But this setup two different possibilities, right? Maybe one possibility is that resting is the best measurement condition. And the other possibility is state challenge is the best measurement condition for getting trade variants. And I drew- I mean, I think in those kinds of situations, where you really don't really know one of my favorite things to do is to draw from the sort of the Platt-Strong inference kind of thing where we you have multiple working hypotheses, and you sort of pre register, if you will, you know, if it goes this way, it's because of that. And if it goes this way, it's because of that. And then we just sort of do- Let the Roman gladiator-
Simine Vazire
Yeah.
Jim Coan
You know, contest begin.
Simine Vazire
Yeah. And actually, this study that I did that I had an NSF grant for. The first draft of the NSF grant, the first three versions I got rejected, had competing hypotheses. I said, Well, here's theories that would predict that the self would be more accurate than others. That we would perceive the fluctuations in our own personality more accurately than our peers would. And here's theories that we predict that actually, we're bad at pursuing these things ourselves, and peers would be better. And the reviewers and the program director came back and said, You have to make a directional prediction.
Jim Coan
Just a single directional prediction?
Simine Vazire
Wouldn't it be better to fund a project that either way, it would be interesting? Than to fund A project where I have to make-
Jim Coan
I'm going to say to the world that that comment was bullshit.
Simine Vazire
So that was a long time ago. And I don't- I hope that we've evolved since then. But I think I literally flipped a coin, because I didn't have an intuition, which one- Which way it would come out. So I don't remember what I predicted.
Jim Coan
The best possible thing to do is to have multiple working hypotheses.
Simine Vazire
Bad pre registration because I made up my my prediction, because I didn't have- I had to pick one way or the other.
Jim Coan
Well, well, you'll have to cross that bridge when you get to it, I guess. So preregistration. Have you published any direct replications at this point?
Simine Vazire
No. I'm one of the many, many authors on a Triple R.
Jim Coan
Yeah.
Simine Vazire
It hasn't come out yet, actually, so no.
Jim Coan
Yeah, I haven't. I don't think I've published. I mean, you know, so many times I've included a resting condition in EEG asymmetry stuff over the years that I guess that counts every time. Even though sometimes it doesn't work.
Simine Vazire
Yeah. Yeah.
Jim Coan
But that was an interesting conversation. So my conversation with John Allen is really interesting. I'd almost love to send it to you and see what your take is. Because we talk about how insane we were to keep going with that measure for as long as we did because we had so many failures to replicate. And we publish them. So this is interesting. This is the other thing about- this John Cassioppo, I don't know if you follow his work much. But he for a while he was editor in chief of the Journal of Psychophysiology. And he made it really plain. This was like 1996. He goes, Send me your failures to replicate.
Simine Vazire
Cool.
Jim Coan
And so we did. John Allen did I should say. Ishouldn't take the credit for it. He published a big, high powered paper of frontal EEG asymmetry that was all a failure to replicate.
Simine Vazire
Yeah.
Jim Coan
And it caused a firestorm. And then, a guy named Dirk Hagaman in Germany did the same thing. And then, we were just- People were failing to replicate a bunch, not just a little bit. I mean, it's sort of, I mean, I don't want to go too far down this road, because there's a whole other set of thing, but it sort of brings to mind this sort of Amy Cuddy controversy. Because there weren't just a single, or two or three, there were several failures. And it was at that point that we wrote our big RO1 that said, Okay, we're gonna get hundreds of subjects. We're going to try and figure this out. And we kind of did. But the period of time where there was this turmoil around that measure was like, almost 10 years.
Simine Vazire
Yeah, I mean, and I think that- I know, this is very idealistic and naive. And I know that humans are more complicated than this. But the people who were the first like, let's say you originally published this successful, you know, frontal asymmetry study, and then this whole debate comes up about whether it's real or whether it's replicable. You should be flattered that that many people want to spend that much time and attention trying to figure out if this effect is real or not. Thats just as important-
Jim Coan
Yeah, right. Right. Right. Right.
Simine Vazire
So I think a healthy science is one where we've identified some really key problems or questions that the field is committed to answering and then we engage in a vigorous debate and adversarial collaborations and try to get to the bottom of it. And it's not personal. But I know we're human. I know we get invested in things. But that's what I would like science to look like. And it sounds like some sub areas of psychology have achieved that. And I just want that to be the case across the board.
Jim Coan
Well, I can't think of a better way to underline your space in our broader scientific world right now. So thanks Simine, that was awesome.
Jim Coan
Okay, that's it, everybody. No more of that, for now anyways. Thanks to Simine Vazire for coming over to my place and letting me record a conversation. She was the first, I think actually, she's the only person I've ever interviewed at my actual homes here in Charlottesville, Virginia. She was in town for the annual meeting of SIPS, which is again, the Society for the Improvement of Psychological Science. You can check out more about SIPS at improving psych.org. Dot O R G as they sometimes say on the radio, and see, you know, for yourself, all the various things that they are up to. You know, turns out that this episode is going to wrap up 2017 for all of us here at Circle of Willis This is our last episode of the year. Which you know, 2017 was... You know what? I don't I just I don't want to complain right now. I just I don't. I had this whole thing here. I was going to complain and I just don't want to do it. This episode, I'm feeling inspired to do something other than just complain. And I don't know what that is right now. I don't know what that's gonna be. What that's gonna look like. Maybe it's, you know... You know, my friend Andrew Sneathern is running for Congress right now. For the House. For Virginia's Fifth District. Maybe that's the sort of thing we all ought to be doing right now. Or supporting at least, right? Right? Let's, I mean, you know, resistance. Maybe let's just vote this year. How about that? Let's just try to fix a few things around here in 2018. Anyway, happy New Year, everyone. Let's not succumb. Let's not do that. Let's keep we got to keep going. Okay. All right, folks, the music on Circle of Willis is written by Tom Stouffer and Jean Ruli and performed by their band the New Drake's. For information on how to purchase their music, check the about page at Circle of willis.com. Don't forget that Circle of Willis is brought to you by VQR in the Center for Media and Citizenship, both at the University of Virginia. And that Circle of Willis is a member of the TEEJFM network. You can find out more about that at teej.fm. If you liked this podcast, how about giving us a little review at iTunes. Letting us know how we're doing. It's super easy. Send us an email, if you have to, if you want to, by going to Circle of Willis podcast.com and clicking on the Contact tab. In any case, I'll see you all at episode nine, where I talk with psychologist, entrepreneur and good old friend Hal Movius of Movius Consulting about the science- Bringing science into the private sector and using science to inform how we negotiate with others. It's one of the most unpleasant experiences anyone can have as a negotiation and has been coaching people on how to do that for a long time. Until then, bye bye.