349 Errington
===

Kevin Folta: [00:00:00] Hello everybody. And thank you for listening to the talking biotech podcast, brought to you by collabora today, happy birthday to the series. This podcast began seven years ago. This week after an arm twisting by Joe Rogan. Now we enter the eighth year with a new episode. Every Saturday. The reason it keeps going is because.

Loyal listeners, download it every week and then share their enthusiasm on social media. So we're consistently in the top 50 of iTunes life sciences in the same league as big productions by BBC or CBS or a larger media conglomerates. So thank you. Thank you. Thank you for your. The funny part is I still get emails almost every week that say, I can't believe I never knew of this before.

I've got a lot of catching up to do all the time. And so something is still [00:01:00] broken. I think we can do this a little better. So I'm going to ask you for a favor. Could you please take a more active role in amplification? So, what do I mean by that? Modern communication takes two steps. It takes content creators, and then amplifiers content creators make the stuff like the talking biotech podcast.

It reaches more people from wheat, retweets shares and, you know, even a conversation at the water cooler. For those of you who are less electronically. Which there's some view out there in science. I know that. So going into year eight, let's take this to a new level. I'll be including some broader content for larger audiences.

In addition to the geeky details we've always discussed, but please retweet, share, put it out there, do what you can to use your conduits in your networks to share this media, because that's how we. And it's important to grow because I think we got a good product. [00:02:00] That's helping people understand the science and the current issues around biotechnology and broader science in general.

So thank you for listening and on to today's episode. Hi everybody. And welcome to this week's podcast. Now, as most of you know, I'm a practicing researcher in molecular biology and genomics, not just a podcast host, and I've made a career around using novel strategies to answer important questions in biology.

And one of the most gratifying parts of the job is when I read a new paper and they cite my. Expanding and even reinforcing those previous findings, it isn't really nervousness about, you know, maybe I was wrong and some of our best work has been born from hypotheses that didn't pan out. So that's good.

It's about seeing the findings add to the tapestry of the understanding of science that good science grows and seeds new avenues of discovery for other laboratories. But over the last few years, there have [00:03:00] been, there has been more and more question. About the reproducibility of some of the high profile work that's been in our best journals and a lot of discussion around maybe our science isn't as reproducible as it used to be.

There's been a lot of questions about the integrity of research and a lot of questions about fraud misconduct and how do we. Go back to a place where, or how do we achieve a place or get to a place where we know we can trust the peer review process or the whatever process we use to ensure that the public understands science, but also that the public trust.

The science that's been found. And so today we're going to talk with Tim Arrington. He's the senior director of research at the center for open science. And we'll discuss this topic. Welcome to the podcast, Tim. Thanks for 

Tim Errington: having me. 

Kevin Folta: Yeah, this is a really good idea. I really appreciate learning more about the center and how it contributes to public trust around science.[00:04:00] 

And let's start out with the problem. What is the evidence that suggests the current system of peer review is broken and that we need to have some sort of improvement in the way that we're vetting scientific findings? 

Tim Errington: Sure. I think of this as kind of a tiered question. So there's some parts that you've mentioned it in your introduction where.

There's concerns about fraud, right? Some people fabricating things and it's hard for peer review to catch all of that. If somebody makes it up, there's some clever people out there. That's not really what the center for open science is concerned with, but it is definitely a concern. I think another thing that you mentioned is mistakes, where humans right as science has done by humans and humans make mistakes.

And sometimes it takes a while to catch those mistakes. So, this is where you get like retractions or corrections come out of that. I think the pandemic showed us that, that that happens with peer review without peer review. Then there's what I think you probably want to get into more, which is this [00:05:00] question about, oh, the studies aren't reproducible and related to that would be that they're not transparent or open enough to understand if you can actually be able to replicate or reproduce what someone else.

And so there's a lot of, there's a lot of growing evidence and in all of those spaces, right, there's a lot of work where one of those, those groups that have been contributing to this I just finished one up in preclinical cancer biology, which is my own background trying to replicate, reproduce somebody else's work and running into challenges in terms of just understanding, oh, what did they do in the first place?

And then of course, when we actually attempted to do it, not being able to actually understand all of the details and the methods and getting similar results was definitely a challenge. 

Kevin Folta: So, is it really boiled down to people not being complete in their methods are not being conscious of, you know, small details that make something truly reproducible.

Tim Errington: I think it boils down to what is the basis for how we communicate and where that incentive is. So [00:06:00] the way we communicate a scientist is through academic papers and those papers. Don't actually focus on what you just said. They focus on what you found, not necessarily how you found it. And so I think it's really, it's a challenge for, for researchers to do that, especially when you're trying to write down methods after the fact, versus just kind of being able to somehow show what we did during the process.

And so methods and papers tend to be quite sparse. They tend to lack a lot of the detail. And I think that is part of the problem. The other part of it is that the journals, right, as you know, in peer review, people like positive, clean narratives and science doesn't look that way. And so there's a, there's a little bit probably unconscious factor going into this.

There's like I said, fraud is the conscious part, but I think the majority of it's unconscious, where as researchers, we're just trying to publish exciting findings. And what we might be doing is, is to starting that view and to starting what we, what we know about how we actually found those exciting. 

Kevin Folta: Oh, I, I agree with that [00:07:00] a thousand percent.

I mean, how many times I've said to people in my laboratory, you know, sometimes putting a paper together is like making sausage, you know, you heed all the parts and all the little snips and pieces and you put it together in a way that works. And it wasn't necessarily the flow that got us from a to B as the researchers, but it's the one that makes the best story.

It makes it best to communicate. To a scientific audience so I could see how things could get mixed up in there a little bit. 

Tim Errington: Yeah. I think you, I think you nailed it on the head, right? Eh, we, we write papers to ourselves, right. Human to human. And so we like clean stories naturally. Think of any novel you write or any other news story.

So I think you're a hundred percent right. And we tend to craft a narrative, not necessarily the way we found it. I think the challenges we get is it's slippery slopes, right? So a good example of that would be, imagine in your lab, you have somebody doing a whole bunch of experiments and then in the paper to make a clean story, you only report part of those, right?

The ones that fit neatly into our narrative and you exclude. [00:08:00] You know, the experiments that didn't quite fit either because they don't fit, which is sometimes how it happens or you don't understand how it fits. And so what happens is you've, you've, you've accidentally distorted the narrative because you didn't actually sharper everything.

Kevin Folta: And I understand that too. And that's something that, you know, of course we wouldn't try to do. If we had some data that didn't fit, we would include those data and say these data, we can't understand how they make these relatable. And a reviewer would probably flag that and say, well, if you can't show how the dots connect, then why are you showing the dots?

And so it doesn't seem just like it's an awful. You know, scientist problem it's with the review system that I want to put it out there because either someone else may interpret it differently or in this. Of their own understanding of the research question. Maybe it kind of, maybe it does, maybe it's the connector, you know, maybe it's something that connects what they do to what I do.

So it does make the question of scientific publishing [00:09:00] really difficult. So how do we fix that? 

Tim Errington: Yeah, that's a great question. And I could not agree with you more that when, when we think about where the the challenge lies, it doesn't lie. I actually don't believe it lies necessarily with the research. I think that's the wrong place to put it.

I think it's the system around it. It's the journals. It's the funders. It's the institutions. So I'll go again from an example, from a large scale, this large scale replication project we just finished and preclinical cancer biology. Some of the challenges we had weren't even around these narratives, per se, they were around just being able to say, what can you show all the methods and the materials, right?

The reagents you used and the data that you found. Can you share that with me now that the paper has been published because it's not published with the paper, write the paper is more of a postcard for what actually occurred. We're asking for the details of what was presented in those figures and they weren't able to do.

Right. The, the researchers weren't able to do it, not because they didn't want to do it. It's because the system didn't support them for it. Right. The institution didn't really support all of them with good data management. For example, [00:10:00] the journalist didn't really, or the peer review system that it really enforced that that had to be shared, even though it could have right.

In a digital age, it's quite easy to share this type of data. And it wasn't being enforced all the time. Or reagents weren't being put in repositories where commercial places, where people could buy and use them again. So the system I think is actually, what's kind of failing and the researcher is just the one that's being kind of stuck.

And so I definitely found that and the project that we did. And so I completely agree with you and it's not, I think that's actually the challenge, which is, it's not just one group, right? It's not just, Hey, journals, change your policies. That'll help. Don't get me wrong. But I think it's also funders reward that.

Right. And make sure you find the ability, not just to do the research, but to make sure that it's open and re rigorous and reproducible at the exact same time and to give the support to do that same thing with the institutions. 

Kevin Folta: Yeah. And I think that the problem with this is that when we see. Stories of lack of reproducibility in the news or when it comes out, you know, just whenever it's, it's addressed [00:11:00] it, especially around things like COVID the folks who don't like the science use that as a, as a cudgel to say, look, you, you can't trust it anyway, because it's not reproducible.

But I think this is the nuance. When I was, I even invented a technique a few years ago, a few decades ago, now that we used frequently and that other people would try to use and could not get it to work. And it wasn't that it didn't work or that it was something that couldn't be done. It worked in my hands every time that we had to have Skype calls where we would do that, we would do the procedure together and I would catch those little nuances or have people come to my.

And spend a couple of days and then we would figure it out. But you know, this idea of reproducibility is, is something that new technology can really help us do better at, especially with respect to methods. 

Tim Errington: Couldn't agree more. Right. And actually, so when we think about right, if something's not reproducible, right, it [00:12:00] fails to replicate the there's.

There's a handful of things that could be at play. And I'm going to again, ignore the ones where it's bad actors, that fraud one. That's definitely there, but I'm gonna ignore that for a moment. You have the originals wrong, right? It's just this very black and white conclusion, the original paper that made that, you know, exciting finding they're wrong, they did something wrong.

They just it's an artifact or you tried to replicate it. Well, you did it all wrong. So that's not true. Right. But I think that actually the, the answers where you just got the more, the more likely answer is they're both. Right. And we don't understand. And that's actually why trying to make our work reproducible.

Why replication, why actually having more openness in it and letting other, other researchers, other people kind of interrogate. It is not to discredit in my opinion, it's to better understand it's right, right on what you said. Okay. Well, I told you what I believed was necessary to do it, but there's so many factors at play.

Maybe we're missing something, so let's do it together or let's document more because maybe I'm focusing on the wrong thing. And we hope that it's something [00:13:00] exciting, right? We hope it's some new, exciting, you know, discovery about the world around us. And sometimes it's not, sometimes it's a technical glitch.

But the only way to do that is to actually interrogate. 

Kevin Folta: And then maybe I can give another example just for, just for the listener, is that there are. Other overlays that vary from one laboratory to the other and meaning that in, in my laboratory and in, in the experiment I did when we harvested those seedlings in those nuclei, right.

At, you know, first thing in the morning, The data come out very different than if you do it in the afternoon. And even though you're doing the same experiment and you're doing the same, everything, there's maybe some sort of circadian influence or some sort of other influence that's keeping the data separate and it never got into your materials and methods because you never said, well, our lights went on and off at this particular time.

And then here's where we did the experiment yet. That may be a important and important gate. Dictates the final outcome. [00:14:00] And why that's important to note is because now we would discover that. That particular influence. If two groups did influence, did interrogate the data at two different times or did gather the resources two different times.

And this would allow us to discover those new edges of maybe some other kinds of influences to why the data were in congruent. And so if you trust both sets of data, it gives us an opportunity to really look at it carefully and say, okay, I trust this. I trust this. Why are they different? And that usually means to answer as much more complicated than we originally thought.

Tim Errington: Yeah, absolutely. Right. And you can keep going on that thread because the way right science, no single finding no study, no paper is definitive. Right? Science is a process. It's not. You can't have some exciting finding, but like, that's it, that's what we do. That's set in over with. It's just a step in the process and you hope that everyone wants and for it to keep building, right.

Somebody else can take it and build it. Right. We, we are, we are [00:15:00] gathering knowledge, but sometimes the best way to go forward is the way you just described it. As you sometimes have to go a little bit backwards. And really what you do is you have to ask this bigger question, which is all right. Well, does the evidence keep pointing towards us on building?

Even if we're taking some steps backwards, are we getting a better understanding of. Both the specifics of what's necessary, but also that it's robust, the variations, right? There are so many variations that in the lab, we attempt to control them, but we're hoping that we can understand something more fundamental than that, even if it is complex or when you keep having more and more groups interrogate that you realize that actually it's very, very specific and the utility of it starts to fall away.

So it doesn't mean that it's not reproducible. The generalizability of it, right? The utility of it has, has started to crumble a little bit just because it's very narrow in the, the, you know, the situation that it's applied for. So take like your example, right? If there's something exciting about the differences there, it helps us understand the overall biology of it.

We can actually learn from it, but if it turns out it's something that actually narrows it and it's not useful anymore, or not as broadly [00:16:00] useful, we might not invest as much resources in it. Again, neither of those tell you its quote unquote right or wrong. It's it's this process that I think is really, really important to.

Kevin Folta: And do you think that the modern publication environment really foster this? Because back when I started in science in the late eighties, you had a conversation that took place in the literature and you frequently saw papers that were three figures that, that just kind of incrementally move the science forward.

And you almost had this conversation that was happening between different scientific groups that would build on each other versus now you have. You know, to publish in a, in a marquee journal, they want 13 figures, 36 supplementary tables. You know, it's such a, they want a complete story now. So instead of having a conversation in the literature, it tends to be like, we drop this entire conclusion into the literature and you know, almost like it is today.

Tim Errington: Yeah, I couldn't agree [00:17:00] more there, there was. Before we kind of answered that I have my own little anecdote from when I was in grad school, I was, I studied clamorous biogenesis, which is really fun. And there's some really cool early papers in the nineties for me. And these are like site and these are the big journals, right?

Science. This was the science nature papers that I'd read. And it was, it was just like, 1, 2, 3 figures, max you know, really elegant assay's, but not a lot of data. It was all about the methods. And I remember reading, even in those journals that the methods, they were, one of them, they were talking about the blender that they were using to kind of grind the tissue, to eventually extract the enzyme.

And it was super detailed. They told you all this info about the blender, right? In order for you to really understand how to do it, you don't, we don't do that. Right. Nothing. Right. We just say spin. You're lucky if somebody tells you to spend something. Right. And so I think you're absolutely right.

We've gone over this idea that. Overwhelm you with information and data that, that sufficient versus recognizing that it is this conversation. And again, remembering that these journal articles are really for the [00:18:00] scientists to just communicate as scientists, that actually the, the details are behind it.

And I think that's something that, especially in the current system of, of the way that we can digitally talk, even right now, we're not leveraging that as much as we shut in research. And I think we need to kind of remember that the publication system was novel at the time. And it's maybe not so novel and it needs to get a little bit more of innovation to be able to really harness the digital age.

Kevin Folta: Yeah, really good point. And how much of a lack of reproducibility is due to fraud? 

Tim Errington: You know, I don't know that question. It's a good one. I secretly hope it's very small. It drives me nuts to think people do this. Because I'm like, why are you going to go do something else? Why are you in science if you want to just make up stuff?

So I don't, I don't actually know that percent. Depends on how you estimate what groups you kind of go to to get those estimates. There's a lot of attempts to have checks and balances on that. And there's also some great people that are doing it. I think sadly it's probably higher than we think because people are quite clever.

So hopefully it's in the sub [00:19:00] 1% range, but I have no idea. 

Kevin Folta: Yeah, I think it is pretty low, but I think it happens on a very consistent basis. I think that there's a lot of with tenure and promotion and the push to publish or perish and with some some countries even incentivizing publication in Marquis journals with financial rewards, it seems almost like it sets it up for there to be abuse of that system.

And I mean, there's like you say, there's some people who are out there who make their lives work. Finding this stuff, but even when they find it, it nothing happens. There's no big apology, no big. Usually these people are fighting back against the whistleblowers. So how much you know, in, in the parlance of public trust, how much, how damaging really is it?

Tim Errington: No, I think it is damaging only in the sense that it shows That the system itself does have these flaws and that we have to be very careful. So I think there's a little bit of [00:20:00] damage that occurs there at the same note. I think actually we can do a better job of really trying to like highlight it, but, but recognize that that's actually the whole point of the scientific process.

It's not a, it's not a trust me enterprise. It's a show me enterprise and it's self-correcting and if we want to be self-correcting, we have to show it and do it out in the open. And hope that that's the process that still does it. I do think it, it shows the ugly side of, of, of science, which is all the human motivations.

You just listed them. All right. It's an Inn, which is by the way, some of it's natural, it's just the bad actors that are going there and kind of skirting around it money. Right. You know prestige, these are the things that actually are driving it. And I think it's a slippery slope again, where. People want to be able to be successful in their career.

If it's not, they're not getting the results, which by the way, as researchers, we do not control the results, we control the questions and the methodology that, you know, the outcomes are what they are, or at least that's how it's supposed to be. I think it's, I can imagine why people maybe go down that route, right?

They're like, oh shoot. In order to get my next grant in order to get my job or keep my [00:21:00] job, I need to get this exciting finding. And I know that this exciting paper, this exciting grant, and the only way to do it is to make a very neat, clean story. And I think you just have. Go too far and just make it the problem that makes it persist though, that you talked about is we as researchers still, we have a hard time correcting, right?

We talk about the literature being self-correcting. But when you see that a paper gets retracted, it still tends to get cited a lot. So again, however much you want to value a citation papers that are retracted still gets cited at a very high rate, which is when you think about it should stop happening, right?

We're supposed to say, oh no, wait, don't cite that paper anymore. But people still. 

Kevin Folta: Yeah, it still happens. And, you know, just to your point, I think it makes for very non-creative science, because I can say this and I don't know if I should be proud of this or not, but we go into so many hypotheses in my laboratory that we do the tests and we get the data back and we go, this doesn't support what we think.

It. And, and so [00:22:00] it means we really under thought the question. And then when we go back and we look at it again, maybe do another experiment or two, we find out that the real answer is much more interesting than what we thought it would be. And so this was really the, that's what we lose when people are starting with a conclusion and going back to.

Tim Errington: Oh, I completely agree with you. Yeah, you don't, you don't want to have the thought that somebody could just, you know, craft their own narrative and make a compelling story as nice. But if it was, if it was that simple, I don't think we'd be doing the research personally. I think you want it to be really hard.

Kevin Folta: Yeah, that's it. It's very true. So we're talking about reproducibility in science, and we're talking about the factors that influence and its effect on public trust in science and how we can maybe make the system better. We're speaking with Tim Arrington. He's the senior director of research at the center for open science.

This is collaborative talking biotech podcast, and we'll be back in just a month.[00:23:00] 

And now we're back on collaborative talking biotech podcast. We're speaking with Tim Arrington. He's the senior director of research at the center for open science and, and the first part of the podcast, we talked about the problem of reproducibility and possibly some of the motivations that come about.

What is the big effect on public trust and where does that manifest? 

Tim Errington: Yeah, I think so to me, the effect on public trust is about how we can demonstrate how the public should trust the research and how it should trust. The research is not where it's being published or the fact that you know, some famous researcher or at some famous institution did it, but it's the process.

They have to be able to trust a process. And I think the key to that is to get back to what is the part that you should trust is that the peer review part. Maybe, maybe not. There is, I think there is a lot of pew has done some [00:24:00] studies to suggest that there is a lot of need to make sure that there's independent review.

I'm a big fan of it as well, but it doesn't have to stop there. I think the trust is also that we, as researchers share right, that we share our data, we share our reagents, we share our methods and that we are able to. Be open to counter evidence, to what we presume we think is going on. Even if it's, even if it's, you know, against our own, you know, models that our favorite pet models, if it turns out that maybe our evidence supports something different, we have to be able to update it.

You were just talking about that before the break. And I think that's the way that the public needs to trust, which is we need to demonstrate those scientific ideals, which is really the process that I think we're all taught early in middle school. Right. Which is what a science club. 

Kevin Folta: That's right. And then one that we still don't build into people enough, despite it being a middle school thing.

And I, the thing that always helps me the most is the thing I don't have to be. Right. But I have to be not wrong. I think that's two very important things. How much [00:25:00] of the process do you think is clouded by things like predatory journal? I 

Tim Errington: think that has a little bit to do with it. Again, the fact that we even have predatory journals is, is demonstrating the problems that the system is kind of imposing on us.

You know, I don't know. I don't know how much that is. I think, I think that's part of the problem. I think the bigger problem personally, for me is more about it. Doesn't matter what journal you're in. It's more about where, what you value. So the fact that we value a journal publication, maybe that's the problem and that allows the predatory journals to exist.

Instead of recognizing that. Well, actually, maybe what we should recognize is, is in addition to the journal publication, is that you shared the data and that other people were able to reuse it, that you shared your reagents and that somebody else was able to take it and move in a new direction. Like those artifacts, we tend not to value as much.

And I think that's part of what's causing this. 

Kevin Folta: So how do we solve this problem through the center for open science? 

Tim Errington: Yeah. So that's a great question. And let me give you a little bit of background of what we're [00:26:00] trying to do at the center. So there's a couple of different activities that we do. And again, trying to recognize that the way that we approach it as twofold one, this is a systems level approach that we're taking.

It's not just one, not just the researchers that we need to support. Our shift. It's also the journals, the funders, the societies. But also to recognize that the center for open science, we're just one of many actors within the space, right? It's a decentralized sciences decentralized. And so that's a really important thing to consider, and there's a couple of different main activities that we do.

What I lead is the research effort. There's a number of, of what we call ourselves as Mehta scientists, scientists. Conducting replication studies or looking at interrogating the way that we both conducted and share scientific findings and trying to figure out well, where is there inefficiency in the system?

How can we improve it? And if we try different ways, are there any risks that come along with. So that's a really, I love it. It's actually been really exciting for me to kind of take a big step back from the bench and to look at it a little bit more holistically, but I can't, you can't just study it, right?

I [00:27:00] mean, you can, but you can't in order to make progress, you can't just study it. You have to be able to also make the change within it. And so a couple of things that we do is one. Is to leverage the fact that, again, we're in a digital tech digital age and there's technology that can support openness and reproducibility and rigor.

So we maintain and build a open source software to help researchers share their data or their methods or their protocols, or do techniques called something called pre-registration, which is basically a similar to clinical trials. Right. You register your trial before you do it, and then no matter what you find you report on those.

So we're trying to encourage these types of rigor and openness practices to try to increase the trust in the science and the science scientific process. So actually enabling it through infrastructure. And again, we're connecting with other groups that are doing the same thing. The other piece of it is also making sure that we work with those incentives.

So again, we're humans, scientists, we're driven by those journal publications. Maybe we can both broaden the scope. It's more than just the journals. It's also the data and other artifacts [00:28:00] that you can share and make open, but maybe you can also align those journal policies with the incentives you want.

So that's great. It's about how creative your research is or how innovative it is, but let's also make sure it's how open and rigorous it is at the same time. So those can be factors that we can also do. So essentially shifting those incentives to align with the values that we, that we want to. 

Kevin Folta: And I really liked this.

So if you have a pre-registration process for an experiment or for, let's just say the pub, the, what will be published, you kind of set it up where you know that the data will be there, whether the hypothesis is supported or not. So does it really provide a venue for us to be able to publish. You know, in quotes negative data.

Yeah, 

Tim Errington: absolutely. Right. So you're going on this right thread. So when we think about the, the main contribution that we do as scientists, it's really as our ideas, when we stop and think about it our, you know, the methodology and the outcomes are just a manifestation of that. So this concept of pre-registration saying, all right, well, this is what I'm going.

This is my [00:29:00] hypothesis. This is my research question. This is the way I'm going to go about doing it. One of many ways. And I don't know what the outcome is, but if I can pre-register that, then I've essentially staked one a claim, which is, this was my idea. This is how I thought about it. So, Hey, everybody, like, look what I did.

Look what, look at my contribution. So I think you were right to say that word publication, because that is, that is a form of publication. It's just on the idea and the methodology of post to the outcome. And then you're absolutely right, which is, it allows broader dissemination of all findings, regardless of whether they're exciting and positive or whether they're.

That color boring, but important mundane part of research, which is. 

Kevin Folta: Well, and very important results. I wish that we had something. I wish that this existed already in a really big, invisible way, because how many experiments are repeated that we would not do because we knew that they would. Either be feasible or would not [00:30:00] yield any interesting data, you know, or would, would we kind of just reinforce the same old story?

You know, the things that we think are transformative that turned out to be mundane and really not worth publication publication, because someone else did it already, or in which case it would show reproducibility, you know, w which would be a real nice thing too. So I really liked this idea. Are there other examples of tools or software that you could talk about, which would.

Enabled through center 

Tim Errington: for open science. Yeah. So another thing that we do, and we're not alone here as well is enabling preprints. So again, if you can recognize that another form of, of communication, including those null results Are submitting them as a pre-print before publication. I think bio archive and met our Khyber, obviously ex excellent examples, but really broadening that to as many disciplines as possible.

Something else that we try to do. And that I think allows what we've seen in the pandemic is faster dissemination and it allows a different form of peer review, which is you can still have that traditional peer review to get you into a journal [00:31:00] for whatever those selective criteria are for that. But it enables a more public discussion over, well, what are these initial findings?

And maybe I can begin to improve that because again, research is an iterative process. And so we enable that as well. Again, having enabling communities to kind of develop their own pre-print guidelines in order to kind of foster that in different areas that, you know, current pre-print communities don't.

Kevin Folta: Yeah, I, I think that's really clever and I, I love it because the reason I like preprints, I like to read them, but I also really like to read them in controversial areas because you find so many of them like around COVID or genetic engineering that they end up in the preprint server, but never are published formally.

And that's really informative. Because it tells us that this did not survive the rigor of peer review. And so there's a lot of other good med information we get from preprint servers like that. So is there utility in preprint servers that actually [00:32:00] help the author? Get more information from the public in kind of pre peer review to shape the manuscript.

Tim Errington: Yeah. There are a couple of ways to do that. So one, obviously it's, it's open access. That's a, probably one of the biggest things is, is the public. Anyone can get access to it on OSF, on the OSF. Preprints we have, we enable a tool called hypothesis that lets people comment on it. So you can actually begin that dialogue.

Right. So. And if, if those aren't familiar with the tool hypothesis, you can basically highlight a specific text within the PDF that's being displayed. And you can actually just start to write comments. You can, you can start to actually recreate this kind of public leisure to, to discuss and communicate with it.

There's, there's one more thing that we do also we've been working on and again, trying to work with others because it's a, I think it's a, this ecosystem approach, which is really trying to rethink how you assess credibility in the first. So preprints, I think really is a great test bed for this. I think normally it's probably perceived on one that's been peer reviewed or two it's [00:33:00] been peer reviewed and it's in this certain journal, right?

This prestigious selective journal, but that's conflating quality with selectivity, which is not really the right way. So instead of you, you, when you put it, preprints, you put it back into the researchers' hands and you can kind of reinvent that system. And so we've been doing some author assertions about, well, was this study mean it was the paper, any experiments where they preregistered is the data open, right?

What is your conflict of interest? You can actually start to have these other indicators of trust that also enable the public to. To kind of get in a conversation about it to start to learn. Actually, I think the important parts, which is, well, how do I know to trust anything? And in the end you do have to start to see these other indicators of trust.

Kevin Folta: No, that's really good because there's a lot that you can clear up ahead of time, especially with conflict of interest. And especially if you're able to clear that up and get out in front of it. It really does separate conflict of interest from misconduct, which are two really important differences that the public isn't clear on.

And I think that's a really, really important point in [00:34:00] terms of how this is all funded though. How, how is the center for open science? 

Tim Errington: Right. So we've been around since 2013. So we're a nonprofit just to make that clear to folks and our funding largely comes from a couple of different sources some philanthropic groups.

So the Arnold ventures Sloan those, those types of groups have funded us and still do fund us the Templeton foundations for. But we also apply for research grants. We essentially look like an academic institution ourselves as well. So we get, you know, grants from NSF or NIH or the department of defense in the U S so we have a lot of research grants that fund us.

And I'd say the last part there is. Individuals who also support us. So we get individual donations, including many people who use our, our infrastructure, the OSF there's a lot of users, researchers they'll either write stuff into their grants or they'll donate to us, which is wonderful because essentially I think it's one of these great systems where we, we very much are.

We see ourselves as a public good service. Back to the research community and the broader public community. And it's [00:35:00] wonderful to see that engagement right back into it. I think of it just like, not just to just, we are this at this level yet, but I think of it as NPR as well, right. A public service and that of people like to use it.

They can help support it and maintain. 

Kevin Folta: All right. Well, let me just kind of maybe wrap up with this. If you could wave a magic wand and create the publication ecosystem that you think would be the best at ensuring trust and reproducibility, what might that look like? 

Tim Errington: Oh, I need a big wand. All right, let me, I'll give you, I'll give you some ideas on this.

So I think one is to disconnect a couple things. I think the ideal system is to allow. Researchers to be able to share among researchers and disseminate those findings without those, those barriers. So preprints essentially really pushing on that as, as the true dissemination of that's, how researchers communicate it.

And that's how we can actually kind of build the system back of, you know, enabling not just the papers, but linkage to all of those artifacts that are important, [00:36:00] right? The methods that everything we're talking about, methods, reagents, code data. So having that base infrastructure to allow the researchers just.

To share with each other freely. So the whole point of preprints, but I think the ideal system doesn't stop there. The ideal system still says peer reviews. Right. It still has a place. And selectively of journalists is not necessarily a bad thing. If you can, if you can turn it around a little bit. And so the way I like to think about that ideal system would be thinking of journal publications a little bit more.

Like we think of news outlets as well, right. Where we can have the same story. Was the same credibility behind it. Right. All the artifacts being shared in different venues. And so we can still use journals. We can still use peer review to help us understand. Well, what kind of storytelling do I like? Or what type of area of research do I like, but we don't let it dictate what gets out in.

To the scientific public domain, right. That that's, that's controlled and that's more freely done that anybody can find all of the content that anyone's [00:37:00] able to kind of harness that and do a lot of exciting. I think Mehta scientists and data science work on top of it because we're freely sharing everything.

Right. We're preregistering our work. We're sharing everything regardless of outcome. And then we use peer review on top of it to help tell that narrative, because again, we're humans. And I think we don't want to get rid of that. So to me disconnecting those two would be the ideal state because it allows, I think those two important parts of broad dissemination, the rigor, no matter what you do, you don't control it as well as that kind of clean narrative that we all itch for us.

Kevin Folta: No, I love it. You know, and the other thing that really just kind of popped into my head on this is that when you look at a tenure and promotion packet in most universities, there's a section for creative works and you could really encourage faculty to create videos around their methods to demonstrate their methods in.

Forgiven paper in a video format on YouTube, or maybe do other types of more extension oriented publications that would enable others to enhance [00:38:00] or to replicate their work and show those little nuances and maybe reveal some of the things that cause lack of reproducibility and that still counts.

You know, you're, you're able to get credit for something. In addition to the publication some other kinds of. Credit. So this, this conver I know a lot of academics listen to this podcast. Think about those kinds of things, because that could really be it's another line in the, in the packet, but it also is something that could enhance the reproducibility of your work.

Ah, 

Tim Errington: absolutely. I'm so glad you went that direction because I think that's exactly the benefit of, of thinking through this, which is, it's not saying get rid of the publications, it's saying open it up because that's the only one piece of it. And this really allows you to demonstrate. The, the rigor and openness of the work behind it.

And I think when we think of early career researchers, This is what we need, we need to do to support those next generation of researchers that are coming in. Right. We, it takes a long time for them to get their papers published in a journal, but there's a ton of work that's behind it that really [00:39:00] demonstrates the rigor of the, what they're doing or what they could be showing.

And I think we need to start remembering that, that this actually starts to open it up and make it more equitable for everyone as well. And then you're right. It diversifies it, and it allows you to tell a different narrative, which. Matt look where I publish my paper, but it's look how much I'm trying to make sure that everything I'm doing is having the maximum benefit it can.

So it's more of this like return on investment question. 

Kevin Folta: No, I love it. I think that's really great. And I hope more universities start thinking that way. And maybe even enabling that with some kind of technical support, video support, that kind of thing, because I I'm with you a hundred percent. So Tim, thank you so much for joining me today.

This is something that I didn't know that I cared about as much as I do, and it is something that going forward, you know, just this conversation really has helped me maybe be a better scientist, but maybe a better. Leader in science to help other faculty find ways to better disseminate their information through these types of outlets.

So if people wanted to find [00:40:00] out more about the center for open science, where would they look? 

Tim Errington: So you can find us at our website that's cos.io. You can also find out more about us at the open science framework, osf.io. And for those on social media, you can follow us at Twitter at our handle. Oh, S F R a M E w O R K.

Kevin Folta: Very gut. So thank you very much, Tim, for joining me today, this is really exciting. And if anything comes up in the future that you would like to share with a broad scientific audience, please get in contact with me again. 

Tim Errington: Okay. Absolutely. Thanks for the invitation. 

Kevin Folta: And of course thank you for listening to the talking biotech podcast.

We really appreciate your reviews. And for you telling friends, there's two sides to this equation. There's me hosting this every week and having great guests who share their stories and their expertise. That's one side of it. The other side of the communication process is you disseminating that. Work and amplifying [00:41:00] the efforts of this kind of content, share it with friends, share it in your social media and help this story get out so that more people can learn about the new ways that we're able to help reproducibility through these open science frameworks as it worked.

This is a talking biotech podcast, and we'll talk to you again next week.