[00:00] Jonny Coates: 
A few years ago, a paper appeared in a major scientific journal with an image of a rat, except the rat had impossible anatomy. And this wasn't a joke or a test. It was a peer-reviewed paper that had been approved by an editor and was published in a scientific journal. Even before the rise in misuse of AI, there were paper mills, peer review cartels, fabricated data in some of the most prestigious journals, and more. This is SIAcast, a show about the science of science, and today's episode is about trust, not trust in scientists as people, but trust in the very system of science itself. I'm your host, Jonny Coates.
[00:39] Yagmur Ozturk: 
And I am your co-host, Yagmur Ozturk .
[00:42] Jonny Coates: 
We also have our guests.
[00:44] Joanna Diong: 
Hi, I'm Joanna Diong.
[00:45] René Aquarius: 
I am René Aquarius.
[00:52] Jonny Coates: 
There have always been stories of problematic papers. These problems may be small and accidental. They may be deliberate, almost infamously so with western blots. There may even be big problems, like conspiracy theory big. A well-trodden example of that is the autism vaccine paper from the disgraced Andrew Wakefield. The impact can also have enormous implications for entire fields of research. In recent years, we witnessed the Alzheimer's field reckoning with a significant case of fraud that has resulted in millions of dollars wasted on dead-end research built on fraudulent data, not to mention the damage done to the careers of researchers. So just how does work that shouldn't pass, pass? The rise of artificial intelligence is making it much easier for paper mills or fabricating results and entire datasets that look so real a peer reviewer couldn't tell the difference. We're now seeing an explosion of papers with problematic uses of AI.
[01:44] René Aquarius: 
You know, it's funny, I'm not a, like, huge fan of AI b- mostly because of the way it is u- being used at the moment. AI seems to be the answer to every problem in the world, and I don't think that is valid, a valid way of looking at problems.
[01:59] Yagmur Ozturk: 
I think about the influence of AI in search a little bit like how I think about the influence of the internet in research. Um, so back in the day when the World Wide Web, uh, first became global, there was a very similar idea that it would solve all problems. Over time, this clearly has not happened. Now, there have been real benefits, uh, with the internet, you know, it's sped up the delivery of information to people and place remotely, but the internet has helped this, has done this in helpful and unhelpful ways. So, for example, information spreads, but misinformation also spreads. So I think of it more like a two-edge sword.
[02:39] René Aquarius: 
I use AI every day in my sleuthing. It really helps me. Uh, I use a software tool called ImageTwin, which uses AI to find duplicate elements between images and within images. The nice thing of that type of AI is that, you know, I have this software tool that does something in a very quick and efficient way, and then I have my own eyes and I can check whether the software tool did a good job.
[03:04] Yagmur Ozturk: 
AI can also assist bad actors to operate at scale, um, and I think about recent stories on international conferences, so the Research Integrity Conference, um, computer science conferences where people ha- have had to reject loads of AI-generated abstracts or AI slop.
[03:20] René Aquarius: 
Of course, see now a lot of problems with, uh, hallucinated references that occur in papers. So on the one hand, you can use AI as a tool to find problems. On the other hand, I think AI is now creating a lot of problems on research integrity issues, and yeah, I think we need to get that in control somehow, but I don't know how because everybody seems to use it as a shortcut to write more papers.
[03:50] Joanna Diong: 
My focus is citations mostly. I work on citations and, you know, hallucinated citations became such a big deal recently because people are just discovering them. Not that they haven't been around, but it became a big deal somehow recently. It's a- again, a confusing thing. Also, policies are ver- they're very different. People don't really know what to do. You can't really ban people from using LLMs, but then, eh, everyone needs to be open about saying that, but if you're open, then it might make your work look bad, whatever. So it's just, again, very confusing situation there. Um, in my case, I- I- I don't know, actually. (laughs) Like, the... So it is making things difficult for us to detect, yes, so image generation, every- anything like that, but you know the red penis that we, uh, we have in the scripts a lot? Yeah. But that type of thing shouldn't pass, uh, definitely. AI slop passes.
[04:53] Jonny Coates: 
But there are other issues, too. Peer review is under immense strain with editors struggling to find reviewers, with reviewers increasingly performing poor quality or light-touch reviews, often due to a lack of time or bandwidth to review properly. Journals are in an arms race to screen out AI slop before sending articles to be reviewed by the overburdened reviewers. Journals have also widely adopted business models that encourage poor behaviors.
[05:17] René Aquarius: 
Yeah, it is a, it is a problem. Of course, the publishers with, um, the gold open-access, uh, publishing model, you know, they can really benefit economically from accepting a lot of papers, even if they're bad papers, because every paper generates APC, so that's the processing charge, uh, for open access that every author needs to pay. So if the number of papers go up, the revenue for the publisher goes up, and of course, I don't think that ev- you know, a publisher is in- in it only for the money because they also sell a, you know, a product. They need to sell a trustworthy manuscript to the reader, but I think if, you know, at this point, there are so many problems that we see that I think that the publishers need to take a step back and...I don't think that there's an infinite growth in this market. We all need to be more careful. We need to be more careful what we read, but the publishers also need to be more careful on what they publish.
[06:17] René Aquarius: 
And I don't think that it's very sustainable to have, like, 10 or 15%, uh, increase every year, uh, in the number of papers that they publish.
[06:26] Jonny Coates: 
Another issue is that nobody really takes responsibility.
[06:30] Yagmur Ozturk: 
I kind of think sometimes that it's just part of human nature that we like to complain. (laughs) Pa- part of it is that, and, you know, g- given a chance, we probably have complained about most things. But I do agree that, y- you know, maybe the feedback that we hear is, is a reflection that there are... that the problem's not simple. Um, it is complex and it doesn't sound like one single person or one single entity, um, is able to fix everything. I think some of the, like, you know, when we've talked about the system level, cultural level, individual level, uh, influences, I think, I think everybody has a part to play in trying to help make the system better, even if not every... even if no one can do the whole job. Everybody still has a part to play, um, whether you're working in government and writing the policies and legislation that influences us, whether you are, um, a new PhD student just starting out, um, and then everyone, everyone else in between.
[07:32] Yagmur Ozturk: 
I think it would be worse if we did nothing than that allows, and we exert no pressure on people who are trying to exploit the system.
[07:41] Jonny Coates: 
Mm-hmm.
[07:41] Yagmur Ozturk: 
I think that would be worse.
[07:43] Jonny Coates: 
So while scientists always had errors and some bad actors, what has changed is the scale. Here's Joanna on why research integrity is important.
[07:51] Yagmur Ozturk: 
Research integrity has always been an important topic, um, although maybe there has been an increasing awareness of research integrity, uh, in recent years. So it at least has been important for a lot longer than we've thought about. So at least, you know, the New York trials happened in 1945 to '46, and ethic panels were really developed and came into effect after, uh, that period, after the trials, and then came the Declaration of Helsinki, which is the primary set of ethical principles for medical research involving human participants that came into effect in 1946. So ethics as a part of research integrity, um, has been around for much longer than recent. Um, but I do agree that there is an assumption that research integrity has, is becoming more important recently, and perhaps this might be influenced by a greater awareness, uh, of the need for trust in science, particularly in recent years.
[08:45] Jonny Coates: 
But there's also a growing recognition of these issues. Here's René talking about what got him into the field of research correction.
[08:53] René Aquarius: 
I did a systematic review together with my colleague Kim Weaver. We were looking into animal models that, you know, simulated this type of brain hemorrhage, and we wanted to know if there were any, you know, treatments out there that were... that had been described in animal models that we could maybe use in patients because in real life in patients, there's not much of a, a treatment with compounds or, or medicines or stuff like that. These types of strokes, these hemorrhagic strokes, they are very deadly, so one-third of the patients immediately die. Uh, so it is very important that we find something that we can treat these patients with in the hopes that less people will die from this condition. When we started, you know, finding these studies, we did a systematic review. So you have this systematic search. The idea is that you include all the papers in the field in your search included in your paper so you can analyze them.
[09:48] René Aquarius: 
But what we saw is that we found, like, 600 papers on the topic, and we basically saw that every paper showed that they used different types of medicine and they were all very successful. So on the one hand, you have like 600 animal studies that all show that they're very successful, and on the other hand, you still, after 10, 15 years, you don't have any, like, real solution for patients in terms of m- giving them medicine that actually work. So we thought that was extremely strange, but, you know, we just had a feeling like maybe something's off, but, you know, you can't write that in a paper. Like, "We found so many papers, but it just feels off." So we thought, uh, we saw, like, a presentation by Elizabeth Bik and then we just thought, "Oh, maybe we can just do something like Elizabeth does. You know, we just look at the images and maybe we'll find some, you know, problems in these images." And within five minutes, we found problems.
[10:40] René Aquarius: 
(laughs) So then we thought, "Okay, either we're very lucky or, um, there are so many problems that we just find them if we just have a closer look at these images." Uh, unfortunately it was the latter and we found that in 40% of the papers that we included in our systematic review, we found these types of image duplication issues. So there was a lot, and that really triggered me to look at it, you know, more seriously.
[11:06] René Aquarius: 
And from that moment on, I started sleuthing on a daily basis, which is now, like, two and a half, three years ago
[11:20] Jonny Coates: 
But there are efforts from within, too. Institutions are increasingly creating research integrity offices and roles. Joanna is one such person with a role within a research institution around research integrity.
[11:47] Yagmur Ozturk: 
I'm a teaching and research academic at the University of Sydney. Specifically, my title is a senior lecturer or, or Level C he- here in Australia. Um, and this is most of my day job, but I also serve a- as what's known as a research integrity advisor at the university, um, which is a role placed under the Office of Research Integrity and Ethics. Um, the main focus of this role is to serve as a... as...... first point of contact for staff or for students to approach if they need confidential advice or guidance on things to do with research integrity, so good research practices, academic rigor, um, all- all these things. And we provide advice informed by university policy and the Australian Code for the Responsible Conduct of Research. We're really there to promote good research practices in a way that's supportive and respectful and confidential, and so that's kind of the- the immediate contact point for this role.
[12:48] Yagmur Ozturk: 
Secondly to that, but also similarly important, is we also engage with other research integrity advisors, uh, and the academic community, the broader community, to give training, uh, and support, so through workshops or through school-level, faculty-level workshops, th- those sorts of things.
[13:05] Joanna Diong: 
And what does research integrity look like to an institution?
[13:08] Yagmur Ozturk: 
In Australia, we have a national-level document called the Australian Code for the Responsible Conduct of Research. Now, the most recent version is the 2018 version. It's a high-level document that's really a framework. So, it talks about quite broad concepts, a framework for responsible research conduct, so that we can ensure research credibility. Now, the code will articulate principles and responsibilities, um, that underpin, uh, the conduct of- of Australian research. Examples of principles include things like, uh, honesty in the conduct and reporting of research, rigor in the conduct and reporting of research, uh, transparency in declaring interests, reporting the methods, data, and findings, fairness in treatment of others, respect for research participants, so on and so forth. Um, that's what we take research integrity to mean. So, there are principles, uh, of operating. But the code also outlines responsibilities, both for institutions and individual researchers.
[14:12] Yagmur Ozturk: 
For example, institutions are responsible for es- establishing and maintaining good governance, good management practices. They're responsible for identifying and complying with the laws and regulations and guidelines. Individual researchers would be responsible for supporting a culture of responsible research conduct, providing guidance and mentorship for good research practices, so on and so forth. Now, all that (laughs) still sounds quite broad, so together with the code, there are what are known as guidance documents that give, um, advice and more, in more specific areas. So, there's a document to provide guidance for managing and investigating, uh, potential breaches of the code, another one for authorship, one for managing data and information in research, another one for disclosing interests and manage- uh, managing conflicts of interest, and so on. This is the- this code, uh, is set at the national level in Australia.
[15:09] Yagmur Ozturk: 
We have 42 universities in Australia, and in most instances, um, the univ- individual universities will take this high-level document and then translate it in- in their local context to what is known as a research code of conduct for that university, um, so that they can kind of speak to more of the local context and things relevant there.
[15:28] Joanna Diong: 
When a concern comes up about a paper, what's really happening behind the scenes? How are cases handled at the institution level?
[15:35] Yagmur Ozturk: 
Uh, the short answer is, I don't know. (laughs) What research integri- integrity advisors do is that we feel concerns from people, um, res- researchers on matters of research integrity, and we- we provide advice. Um, now, these issues can be quite broad, and people could- could come to us on research conduct, on authorship issues, research conflict, so on and so forth. And we're really there to provide advice to people. Based on the information we hear, um, in most cases, or in some cases, it's possible to resolve that concern, uh, locally. Like, it doesn't need to go further than the advisor you're speaking with. However, if we hear from, um, the person information that suggests that there's a potential breach of the code and indicates that it might warrant formal investigation, then this is handed over to the Research Integrity Office, and their offices take over any formal investigation.
[16:33] Yagmur Ozturk: 
So, research integrity advisors aren't involved in the formal investigations, uh, of, say, you know, an allegation or a complaint, because that can be quite complex, and it's important to hear all sides of the story. Uh, and we're not involved at that level, but we do provide advice on how people can... concerns of research integrity that they might have.
[16:51] Joanna Diong: 
So, how are research institutions responding to integrity issues on a larger scale?
[16:57] Yagmur Ozturk: 
Most universities would have ethics boards, uh, that have to review proposals and vet them for research on humans and animals. Um, from a governance angle, most universities also have Offices of Research Integrity and Ethics, um, to provide formal support on these matters. And the ethics boards would sit- uh, would be overseen by these offices. Certainly, at the university that I work in, there is mandatory training for staff and students to raise awareness, uh, so that people are aware on research integrity matters. Um, and then at the national level in Australia, we have what is known as the Australian Research Integrity Committee, or ERIC. Um, this is a committee that was jointly established by our two largest national research funders, so the National Health and Medical Research Council, or the NHMRC, and the Australian Research Council, or the ARC, establish ERIC, uh, and administers it.
[17:53] Yagmur Ozturk: 
And what ERIC does is to review institutional processes to manage, uh, and- and investigate, uh, potential breaches of the code. So, we at- at least have some degree of national-level oversight.
[18:05] Joanna Diong: 
But why are people engaging in such questionable behaviors to begin with?
[18:08] Yagmur Ozturk: 
The- the way I see it is the broad aim of scientific inquiry is to generate knowledge to get to the truth. Like, we wanna find out how the world works. But science is done by people. The knowledge doesn't just generate itself. It's people who generate the knowledge, and people can be complicated. Um, some people might be closer to true altruists. You know, they will expend large amounts of energy to, uh, to find the truth rigorously at a- at cost to themselves. Um, some people might be on the other end. You know, they-They might see research success or prestige as the goal, and then work towards getting the potentially bypass or lesser resistance, um, even if it means conscious or unconscious degrees of compromise. And then we have people, you know, in between. Um, they might be the people who are genuinely try- genuinely trying to do their best, um, make genuine mistakes which are only discovered and rectified later. You know, what is the cause of the problem? Uh, is it ...
[19:05] Yagmur Ozturk: 
If we start to ask questions about research integrity, we, we really n- need to think about what drives people, what drives human behavior, you know, where do moral standards come from? How do we operate? What does it mean to be human? Those sorts of, the nature of being human kind of questions need to be touched on.
[19:22] René Aquarius: 
I think there are many, many problematic aspects in academia. I think we're all aware, you know, uh, universities maybe want to compete with each other. They want to be involved in university rankings, which is something that you can game. You can make sure that the people in the labs, uh, in the departments of the university publish more because you put pressure on them. The same you can do in systems where, for example, it is very important in order to progress your scientific career, for example, when you do a study, if there's a requirement that you have to publish scientific work in a Western journal, if you don't have funds, if you don't have the experience, if you don't have a mentor who, you know, supervises you and you are required to do it, it is, I think, very easy to buy a paper online from a paper mill, because that will solve your problem, because otherwise you cannot really address the issue at hand. Same with the H-index.
[20:26] René Aquarius: 
In some countries, it's very important if you want to gain tenure as a professor, if you want to gain ... Uh, if you want to get a research grant, uh, your H-index needs to be of, you know, this and this level. These are all factors that can be gamed. So I think they are not working as intended and they just put pressure on people. And it will be especially problematic when certain people are really gaming these metrics. For honest researchers, it is very difficult to stay on top of things, because on the one hand you have people maybe cheating a little bit and getting higher H-index, while on the other hand, as a honest researcher, you cannot get your H-index up that, this quickly, so you will probably not get a research grant. So I think it is problematic on many different levels, but also that the honest, most honest researchers are probably not getting the funds that they need to do their work, while these people should be supported.
[21:23] Jonny Coates: 
There's also been a community response from the sleuths. Collectively, they have devised a set of guides.
[21:28] Joanna Diong: 
COSIG, the Collection of Open Science Integrity Guides, and it's an open collection of meta-scientific guides enabling anyone to do, uh, forensic peer review or post publication peer review. Uh, with these guides, we try to make it simple for everyone and make the information that is usually hidden accessible to everyone so they can, uh, read the papers and understand and see what sleuths are doing, and then they can, uh, do this on their own. Because lots of information sleuths are, uh, using to detect problems is actually not very accessible online. And we want people to use this, uh, as an authority and put these as evidence to what they're saying in their comments on PubPeer, for example.
[22:14] Jonny Coates: 
How many guides are there? What are the different guides covering?
[22:17] Joanna Diong: 
Currently, we have 34 guides. Uh, we started in June, uh, 4, 2025, and at the time, it was 27 guides. So we grew a little bit. And these guides are written by, by 22 different experts, and they are in five different categories. So we have basics such as, "How do I get started with post publication peer review? How do I even contact the publishers?" We have, uh, lots of email addresses that people can use and we update these regularly. General guides also, such as stealth corrections, the vertical line test, any type of image and citation integrity issues that can be fi- found in here, and more specialized guides such as biology and medicine, material science and eng- engineering, and mathematics, statistics, and computer science guides.
[23:07] Jonny Coates: 
And how did all of this actually come about?
[23:09] Joanna Diong: 
Initially, COSIG started with the leading of, uh, Rhys Richardson, uh, who is a meta scien- meta science, uh, postdoc at the moment in Northwestern University. My project, my funding project, is called Nanobubbles. It's a European, uh, research council project, and we organized, uh, a symposium gathering lots of sleuths from different countries. And the idea, uh, initially sparked there. And Rhys was, uh, talking to a bunch of people, collecting their hidden information about how do they do s- sleuthing. And he was, uh ... He, he did this excellent job, uh, collecting all this information together. And since before, a couple of months before, uh, this Co- COSIG was initialized, he, uh, hired me and Sholav Pirelli as maintainers. By hiring, I mean we're volunteering, of course. And we started editing the guides, curating them, ta- talking to more people about these guides and just constantly expanding the guides.
[24:14] Jonny Coates: 
Yagmur mentioning all of that there, something about stealth corrections. So what are stealth corrections? Here's René talking about his contribution to the COSIG guides.
[24:24] René Aquarius: 
So a stealth correction is, you know, if a paper has a problem, uh, and it gets corrected, what usually happens is that the publisher also publishes a correction notice that sh- says like, "There was an issue with this image and now this image has been replaced in the PDF and this is the new image." You know, it basically describes why the image was replaced. But when a stealth correction happens, then only the original PDF has been changed. For example, the image has been changed, but there is no correction issue published. So it is impossible for people who are reading that paper for the first time to know ...... that there was an issue with that paper. I think this is extremely problematic. Okay, one thing is that it's very difficult to find out. We have now identified, I think, something like 162 stealth corrections in total, which is not a lot if you think how many papers have been published.
[25:18] René Aquarius: 
The only issue that I have is that, in principle, every paper that has been published could have been stealth corrected because you just never know. And I think this is a principle that is extremely dangerous, because you just don't know if anything has been changed in a paper. And I'm not talking about, you know, you know, deleting a double space or, you know, changing one word. Uh, these can be, like, important stuff, um, like images, data, uh, affiliations of an author. Even complete authors, uh, can be removed or, uh, included in a paper without anybody knowing. I just think it is extremely weird that this can happen, and also that some editors, editors-in-chief, and publishers facilitate the- these types of changes in the published literature. I think they should stop. And that's what we hoped by publishing the paper, that we first raised attention for the issue, that it may be not as rare as people might assume.
[26:26] René Aquarius: 
And second, by also now writing the COSIG guide for people, to make people aware that they can look for stealth corrections and also report them on PubPeer, uh, if they want to.
[26:38] Yagmur Ozturk: 
Well, a stealth correction even became a problem for COSIG, because I was, uh, checking something from the vertical, uh, line test guide, and I noticed that one of the examples we used, uh, as not passing the vertical line test actually has been changed, uh, in the paper that we were giving it as an example to.
[26:59] René Aquarius: 
That is crazy. Yeah, I didn't know that. Extremely painful, and th- this is basically often the only way you find out about these changes, because you have, you know, in the COSIG guideline, you have the original image, and then for some reason, you check the, the paper again, and you see that the image has changed, and then you think, "What happened here?" Again, the first time that happened to me, I doubted myself. I thought I made a mistake. So then I looked up the original PDF that I still had on my hard drive, and I checked it, you know, side by side, and I said, "No, the text is all the same." It's just the image that has changed for some reason without any correction notice.
[27:38] Jonny Coates: 
One of the problems with this being such a grassroots effort is that in responding to problematic cases, sleuths can sometimes be on the receiving end of researcher frustration.
[27:48] René Aquarius: 
One time, I did get an anonymous email that was a little bit threatening. I just saved it and decided not to, you know, do anything with it. Uh, I also tried to use as neutral language as possible, uh, both on PubPeer, on my LinkedIn, where I also post a lot of cases on a daily basis, and, uh, to the publishers, because, uh, I'm not doing this to get into fights with people or to get in- into these endless arguments on why people are doing this. I just want to flag bad papers, and I hope that publishers try to resolve the issues that I find. That's the only thing I want.
[28:32] Yagmur Ozturk: 
Yeah, other than the occasional LinkedIn trolling, I guess you have been, uh, in a good position. (laughs)
[28:39] René Aquarius: 
Yes, but the LinkedIn trolling I can handle. Uh, you know, people can be critical of me. That's totally fine. I can answer them if they're respectful. That's fine as well. If I have the feeling that people are, you know, trying to rage bait me or something, I just don't reply, that's fine. (laughs)
[28:59] Jonny Coates: 
And an extra problem for sleuths is that it's hard to know exactly why the problem they've identified actually occurred in the first place.
[29:05] Yagmur Ozturk: 
Also, it's hard to decide on, uh, when it's a fixable mistake that you can tell the authors and when it is not, when it's blatantly a big problem in the paper, uh, uh, that they shouldn't have the chance to fix, uh, you know?
[29:22] René Aquarius: 
Yeah, and, you know, sometimes really big problems arise from really innocent things, and sometimes, you know, problems that look like they are minor things might hint to bigger problems. So, I also don't think that it's sometimes so easy to decide whether a problem is a major issue or a minor issue. I, I just see it now as it is a problem. Can we still trust the contents of this paper? Even though the image that I find might not affect the conclusions, I don't think it's as simple as that. I, I don't think it's very relevant if the conclusions are affected or not. I always think about it as if you, you know, try to buy something secondhand from Facebook f- by somebody, and if they lie about some detail, even though it's small and even though it may not affect the quality of the product or the price, you might doubt, you know, whether you can trust this person or not. And maybe it will incentivize you to stop the deal from happening, even though the lie was maybe very minor.
[30:31] René Aquarius: 
So, what happens in real life, in, you know, these deals through Facebook, uh, everybody would say, "Yes, don't, you know, don't do business with this guy." But for some reason, when it comes to scientific papers, we always give a lot of leeway to authors to, you know, correct mistakes and stuff like that, be- because, in the end, we don't really know where these images come from. Many images don't have any metadata. You cannot really check, you know, if a microscope in their lab has produced that image. We don't know anything, and if we find an issue in a, in an image, and then authors correct it by just saying, "Well, here's a new image. This one is the real one," how can we know it's true? I always find that very, very difficult to process in my, in my brain.
[31:16] Jonny Coates: 
And how are researchers perceiving these kind of efforts? 'Cause-... a lot of people, when you- at least on social media, a lot of people feel like it's split, right? It's either, "This is a great thing," or, "Well, you're attacking me and I'm not happy about it."
[31:28] Jonny Coates: 
Yeah.
[31:29] Jonny Coates: 
Which is usually people who've done something wrong.
[31:31] Jonny Coates: 
Yeah, true, yes. Um, it, I, I agree that how, how these efforts to improve trust in science, h- how they're perceived, I guess, can be different maybe depending on which side of the coin you're on. I, I tend to find that individuals who are new to the process, so people entering research or entering academia, tend to see this, the training and the raising awareness and all these things as, as being helpful. Um, academia can be a complex place to navigate, and it's got a lot of complex processes. So particularly if the training or, or these endeavors gets them at the right time, so the timing works out, it does tend to be viewed positively that, that we're, we are trying to assist them. Um, on the other hand, um, we also do acknowledge that the research environment is competitive. There's pressures. There's a lot of external pressures that can be make very complex.
[32:25] Jonny Coates: 
So yes, we, you know, the jaded researcher who thinks that, "Oh, this training is just another box I have to check off," does... uh, we do come across them as well. And part of it, you know, is, yes, we do have to comply with mandatory requirements. Um, but I think on a, on a, on a bigger level, maybe on an individual level, a lot of it does come down to, like, on the day-to-day, how was research conduct performed. And I think you, if you, if you're, if you've ever been on the receiving end, uh, of receiving, um, queries when there have been complications or conflicts, then, then it starts to hit me and realize that actually a lot of the things that we do that might seem like, you know, just another box ticked can have a role in influencing, um, things further downstream.
[33:10] René Aquarius: 
But community is important. I think that it's extremely important. I, I'm living proof of it myself...
[00:00] Speaker: 

[33:16] René Aquarius: 
... in the sense that I got inspired after seeing a lecture from Elizabeth, uh, who inspired me to look more closely at the images, and now I'm doing it on a daily basis. So, thank you, Elizabeth, for that. So yeah, I think it is important to see that, you know, people are actively trying to basically make the scholarly literature better and better every day. I also think it has, it has a possible downside because there's so much attention for it now. Maybe scientists who aren't aware of these types of issues, or maybe the general public, maybe ev- people think like, "Oh, you can't trust scientists. Uh, can you trust anything that scientists do at all?" So, I think you need to be very careful in how you word things, um, and how you talk about these issues. So it's not like all papers are bad or all scientists are bad, or you can't trust the literature. You know, I think we have real issues, but I don't think that science as a whole is a problem at the moment.
[34:23] René Aquarius: 
But I do think that we need to put a lot of time and effort in this problem right now in order to solve it or make it at least a m- a bit better, uh, for science to be okay in the long run.
[34:40] Speaker 9: 
(upbeat music)
[34:44] Jonny Coates: 
So, we talked about the CoSig guides. Why are they such an important resource?
[34:47] Yagmur Ozturk: 
Because it's so hidden. So normally, especially when you do peer review, I think people don't just expect to come across issues. You look at the creativity of the work, the novelty, uh, if you s- notice anything extremely weird, maybe it, uh, brings up something in the reviewer, but I think usually people just don't start, uh, s- uh, looking at the paper and saying, "Okay, I'm gonna find misconduct in this." (laughs) And it is actually so easy to detect these type of issues. Uh, but if you don't know, then you won't look at them. So, we, we just wanted to make this more accessible to everyone, and that's it. And it can be used by anyone. We write in a basic language. And, uh, we really want editors, reviewers to look into these also. It's not just for sleuths or people who want to be sleuths. It's for anyone who's involved in the scientific publication process.
[35:45] Jonny Coates: 
One of the things that came out in the interviews is this tension between sort of responsibility within the system. This is something... I was at, um, OASPA a couple years ago, and I, I was giving a talk on like, it was on like the final day. So I, I'd had the whole conference where I was speaking to people, and I changed my talk because what I f- what I found was depending on which stakeholder, which group I spoke to, they were effectively just blaming everyone else.
[36:08] Jonny Coates: 
So I, I opened my talk basically blaming as many people in the room as I possibly could-
[36:12] Yagmur Ozturk: 
(laughs)
[36:12] Jonny Coates: 
... for all the problems that exist.
[36:13] Yagmur Ozturk: 
(laughs)
[36:13] Jonny Coates: 
It went down quite well, actually. But the, I mean, the point is, we have institutions who will blame journals or funders. We have journals who will blame institutions. We have funders who will blame journals. Nobody seems to take responsibility for these kind of issues. So where do you think that responsibility lies? Is it with one person? Is it a coalition? Is... How do we fix this?
[36:32] Yagmur Ozturk: 
It's just the whole scientific publication ecosystem, um, (laughs) again, I, as I'm writing my thesis right now, it just, sometimes I find myself, "Oh, this is because of this. It's the incentives." Then it go, boil, it goes down, it connects to something else and then something else. So, there's just, it became such a convoluted ecosystem, such a big chain. Everything is dependent on each other. So, it's so hard to just pick one and say, "Okay, th- you are the problem." I guess the biggest thing would be the economics, financial side of things, like, how much money publishers are making from this and that's hard to change, right? That's why we need the pre-prints, we need the movements that we're doing. But again, then people need publications to rise up in the ranks, et cetera. It's just a very, very mixed situation.
[37:22] Jonny Coates: 
René mentions that, you know, h- h- in his mind, the focus of sleuthing type activity should really remain on-... the accuracy of the scholarly record rather than determining, you know, if a problem arose from fraud or error, so not, not really focusing with the underlying issue. Where do you stand on that? Because that was something I f- I, I always kind of oscillate on this one. I think it should be about the scholarly record, but also, if we don't hold people to account, so if we don't say, "Y- you know, you're committing fraud. You probably shouldn't be allowed to do science," then those problems are just gonna keep happening. So where, where do you stand on this?
[37:54] Yagmur Ozturk: 
Yeah. Again, what René was saying, exactly that, and also he's just, uh, highlighting that it's impossible f- uh, for us to know if it's actually fraud, if it's actually misconduct, even if there are glaring evidence that it could be, it's probably that, but we can't say. Then we will get into trouble. Holding people accountable is extremely important, but I know not many countries will do this, and especially when it's such a, uh, when it's so involved in the cu- culture of science, scientific practice in certain countries, it's just hard to separate good from bad, you know? And it's, it's difficult (laughs) . I just think it's difficult, and, uh, in my case, I usually advocate for looking at the scientific content of the papers. What are the citations? What are the problems inside this paper without actually looking at the authors' names or affiliations?
[38:49] Yagmur Ozturk: 
That's how I've been doing this, but, uh, recently I collaborated in a, uh, work about authorship for sale practices in, uh, in conference proceedings, so we did need to look at the affiliations of the people, like how are the, how are they citing each other, et cetera? And it's also giving so much evidence about, uh, these type of problems. Yes, the content is terrible, but also, there's clearly people who are using these services a lot with none of these patterns of people. So you can't ignore it, but I think making your case about the scientific content is always better.
[39:29] Jonny Coates: 
You know, one of the things that very, very regularly comes up in these kind of conversations and anything reform-type conversations is that we talk about systems, and actually both Joanna and René described things as sort of system level, and I think actually we forget... In doing that, we forget that systems are people. People create the system. People maintain the system. People stop the systems from changing. So how... what suggestions would you have for maybe, you know, PhD students out there or early career researchers who are about to enter that system? What could they do as people to help change things?
[40:10] Yagmur Ozturk: 
I think the most important thing is to raise awareness. I feel like I, I don't like telling people just to, you know... You should look at, into this. I know people are not working on this. How can you tell someone whose main job is mathematics to just focus on issues in the mathematics field? They need to do novel research. They need to find new things to, um... like PhD students. So I am in a weird position in my lab as well because w- I'm surrounded by amazing computer scientists and I'm not doing anything new. I am only looking at published literature, and I'm correcting things. That's my job. And I've given lots of talks. I've talked to many people.
[40:51] Yagmur Ozturk: 
I managed to get some attention from people, especially regarding citation-based issues because it's becoming a big deal in the computer science field as well, but I think what we need to do is just talk to as many people as possible, make it interesting, and make it not sound like we're just hunting people down because that's not the case. We just don't want bad science to come out. That's it. Um, I just want people to care more about this, frankly (laughs) .
[41:20] Jonny Coates: 
(laughs) When things do get through, like that rat penis thing, it cause a lot of damage to trust in science, it... And yet, for some reason people don't seem to relate that back to peer review failure. They relate it back to, well, that, that individual journal or those individual editors, those individual peer reviewers. So what do you think needs to change in the system? Do we need to keep peer reviewing everything? Do we need to create a new trust system instead of peer review? Do we need to do something completely radical and different? What, what, what do you think we need?
[41:46] Yagmur Ozturk: 
Great question. So first, first thing I want to say is, for example, I think it used to be a bad practice to cite pre-prints, for example, because they're not peer reviewed, but I trust my judgment more than peer reviewers. I don't know who did that review and got published in a peer-reviewed paper. So if I'm reading the paper and if I think it's good, then I will cite it. Uh, so I think everyone just should be very careful about what they cite and what they read, and you shouldn't cite anything you're not reading, definitely. So that's my main thing, uh, in this case. Second, I don't know how we can change the peer review system because it's just so... Again, it- it's seen as the quality control, the quality control, and maybe it used to be good when there wasn't millions of papers getting published, but it's definitely not enough right now.
[42:40] Yagmur Ozturk: 
Uh, what I think that could help if we're not going to change the whole system is, uh, detecting some things automatically and flagging them to the editors and reviewers could be really helpful. There's no reason to not detect tortured phrases at this point. I think reviewers and editors should be very careful about these very, very easy to detect things. And then, uh, the rest, uh, can go to the peer reviewer. It's, again, not the perfect system, but I just don't have a better idea of how to make things work. I think everyone who's reading the papers should just be very careful, but of course, that's not solving the situation of, uh, someone who's not a scientist trying to understand a paper or anything like that. And people just believe scientific papers because they think, "Oh, this is very high-quality paper because it's been published." But this is not the case. So I think we just need to raise awareness about issues and just, uh......
[43:39] Yagmur Ozturk: 
hope that people are more careful about what they (laughs) read and what they cite. I would like to hear your idea.
[43:45] Jonny Coates: 
I think about this all the time. It's very surprising how many people cite work that they have not read, and there are constantly questions about, "Can I cite pre-prints still?" When you read a paper, surely you're reading it critically and you're making a judgment on it, and it's su- I... That is one of the things that surprises me most about science, is how few scientists actually do that for some reason. I, I'm, I'm currently writing a few things around what I think we need. So, what I'm currently trying to design is a, a trust system that peer review is part of it, so that, that kind of gets away from... moving away from peer review. We don't need to get rid of it. It becomes folded into something else. But basically it's, it's based around the idea of having multiple different trust signals and indicators for an article, and it's very public-facing so that it benefits everyone reading the article, and I... My approach is to split it into two. So we've got...
[44:31] Jonny Coates: 
I call them static and dynamic signals. So static ones are things that the authors can do. Transparency, right? Making your data available, your code available, all those kind of things. Maybe pre-registering your work if it is relevant, that kind of thing. So basically at the point of publication or pre-print posting or sharing a dataset, whatever it is, that signal tells you how much we can... how much trust we can place in what the authors have done. So if authors have not done anything transparently, I think it's reasonable to say we don't have as much trust in them as compared to authors who've done everything transparently. And then the dynamic ones, they are things that are context dependent. They change over time. They're reflective of how the work is being received. So, context-dependent citations. So, is this work being cited well, or are people saying that there's a problem? Are there pa- peer comments? Are there comments ar- raising clear issues?
[45:21] Jonny Coates: 
Have people replicated the work or bits of the work and all those kind of things? Peer review would be one of those dynamic indicators. Altmetric attention, social media attention, and sort of, again, the context of, of what that is saying. Has it been covered in news articles and stuff like that? Because the more attention something gets, the more likely it is that an issue will be raised. So I'm... What I'm trying to do is reframe it not as... There's no individual signal. It's a collective of things together, but also it's not, "Can we trust it?" It's, "How much trust can we place in it?" So it's sort of a scale of trustworthiness rather than a, a binary yes/no or the, I mean it.
[45:56] Yagmur Ozturk: 
No, that, that's an excellent... Yeah. That's an excellent idea. And yes, that's absolutely right about the data and code availability also. That's the least we can do. I hate seeing like, "Uh, it will be available once published," while I'm reading the published paper. Then where is the code? Where is the data?
[46:16] Jonny Coates: 
Now, this might all feel like systems and institutional level issues, but what can you do as a listener to restore some trust in research?
[46:24] René Aquarius: 
The thing that I would really like to see is that people feel responsible even though something might not, you know, fit in their wheelhouse. I think research integrity in general is something that has many stakeholders. So you have, you know, researchers, you have institutions, you have funding agencies, you know, all these different things that come together, and it's very easy for one to say, "Well, that's not my problem. It's your problem." And if we all just point at each other, then nobody's actually solving things. So, I think sometimes it's just good to say, "Okay, maybe I'm not directly responsible for this, but I'm, you know, stepping up to make sure that it will be better." I think the Sloots are a very good example of this, because all these papers, we didn't publish them, we didn't write them, but we're still looking at them every day in order to find problems and to flag those, because we think it's important for science.
[47:19] René Aquarius: 
I think more people should have this mindset to just, you know, sometimes make something that's not your business, make it your business.
[47:26] Yagmur Ozturk: 
I think the most important thing is to raise awareness. I feel like I, I don't like telling people just to, "You should look at, into this." I know people are not working on this. How can you tell someone whose main job is mathematics to just focus on issues in the mathematics field? They need to do novel research. They need to find new things to, uh... like as PhD students. So I am in a weird position in my lab as well, because we're... I'm surrounded by amazing computer scientists, and I'm not doing anything new. I'm only looking at published literature, and I'm correcting things. That's my job. And I've given lots of talks. I've talked to many people. I managed to get some attention from people, especially regarding citation-based issues, because it's becoming a big deal in the computer science field as well.
[48:16] Yagmur Ozturk: 
But I think what we need to do is just talk to as many people as possible, make it interesting, and make it not sound like we're just hunting people down, because that's not the case. We just don't want bad science to come out. That's it. I just want people to care more about this, quite frankly. (laughs)
[48:39] Jonny Coates: 
Science has always depended on trust, trust that experiments were done, trust that results were reported honestly, trust that somebody checked. That trust isn't gone, but it's under pressure, and the future of research may depend on whether the system can change before the trust runs out.