Open Access
Issue
Int. J. Metrol. Qual. Eng.
Volume 11, 2020
Article Number 1
Number of page(s) 9
DOI https://doi.org/10.1051/ijmqe/2019016
Published online 07 January 2020

© M. Estienne et al., published by EDP Sciences, 2020

Licence Creative Commons
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1 Introduction

In France, like in other countries, research integrity and quality is an important, but sometimes controversial issue. Although, nearly 10 yr ago, the scientific community pinned down fraud (fabrication and falsification of data as well as plagiarism) and other problems with integrity (“questionable research practice”, “sloppy science”, “grey zone”) [1], scientists are not eager to admit this problem. The general opinion is that the system regulates itself.

In France, Inserm [2] is active with respect to research integrity and quality since many years. In 1998, a Delegate to research integrity (treatment of allegations of scientific misconduct) was nominated. Quality management is not compulsory, but is strongly supported by the President of Inserm and the General Directorate since more than ten years. Today, trained quality managers are present in one third of Inserm Units. This number is steadily increasing. Quality-management-tools are used to foster integrity and to manage specific actions [3] and a specific training program for research integrity and quality [4] was designed; the nomination of quality/integrity managers is under discussion.

With the collaboration of eleven Inserm Units (U1026, U1028, U1032, U1060, U1064, U1089, U1098, U1219, U1232, U1238, UMS 16) in four different cities in France (Besançon, Bordeaux, Lyon and Nantes), integrity and quality was shown to be a true issue for the scientists and that scientists are eager to improve their capacities. At the same time, the understanding of research integrity and its link to quality management should be improved by specific actions.

2 Design and conduct of the study

Given limited work force and time available, the study was designed for about 10 Units to participate. For this reason, the call for interest was only open for one week and Units were included on a first-come-first-serve basis. We invited all Inserm quality managers by e-mail to express interest of their Unit in a short study with the aim to quantify the knowledge and commitment of scientists to research integrity and quality. The only prerequisites were the presence of a quality manager, the motivation of the Unit (e.g. response within one week) and the support of the head of the Unit. After one week, 11 Units had expressed their interest and were included in the study.

The study is composed of three actions.

  • First, a two hour lecture on research integrity and quality management with a survey before (T0) and after (T1) the lecture in each unit.

  • Secondly, the use of a “RIQ-box” (Research integrity and quality-box), materialized by a dedicated and anonymous internet space (https://www.google.com/intl/fr/forms/about/), was available to each participant.

  • Finally, the quality manager in each participating Unit audited samples of figures or tables of arbitrarily chosen publications, in order to measure the time necessary to identify precisely and completely raw and treated data, information on data processing, analysis, experimental conditions etc. that are part of the article.

The study was conducted over a three months period in each particular Unit. It started with action one: a lecture given by one of the authors and two surveys, which were managed by the quality manager of the Unit.

The authors collected the surveys, with the help of the quality managers, checked the RIQ-box with a dedicated, anonymous e-mail address (enqueteriq417@gmail.com) and performed the audits.

3 Results

The results of the three different actions are presented: initiation to research integrity, use of a Riq-box (collection of suggestions, remarks and observation of questionable research practices) and audits of the architecture of filed data.

3.1 Participants in this study

All participants are members of an Inserm Unit. Participants were first asked to identify themselves as researchers, Ph.D. students or technical staff. The item “others” means personnel under a different status, such as hospital or university personnel who do research. In the further study and in the text the generic term “scientist” was used.

Synthetized data of all 11 Units are presented here, since the objective of the study was not to compare the situation between different Units.

Figure 1 shows that participants are mostly researchers (senior and Ph.D. students) and about one third (31%) technical staff. As compared to the proportion of researchers (46%) versus technical staff (54%) at Inserm (Report and key figures, 20171), researchers were overrepresented at the lecture and in this study.

After the initiation lecture to research integrity and quality, the majority (72%) wishes to go on with the study and starts with actions 2 and 3 (data not shown).

thumbnail Fig. 1

Participants in the study.

3.2 Action 1: Information about research integrity

Fifty two percent of all participants are not initiated to research integrity, 48% had received some initiation (Fig. 2a). Fifteen percent of those who were initiated had attended a course (>5 h), whereas 85% had only attended a conference (1–5 h) (Fig. 2b).

The survey before the lecture revealed that 56% expressed the desire of a training course, 37% did not wish further training. After the lecture, 92% of participants declared that the presentation had “clarified (their) vision”, 6% declared that their vision had not been clarified and two thirds considered that the lecture was sufficient, while one third found it insufficient with respect to their needs of information.

thumbnail Fig. 2

(a) Proportion of trained/untrained scientists. (b) Prevalence of short training in research integrity

3.2.1 Understanding of research integrity

The specific understanding of what research integrity means was checked by comparing the concepts and keywords the scientists of Inserm associate with research integrity and concepts, before the lecture (T0) and, after the lecture (T1), keywords of the different statements (Singapore Statement on Research Integrity, 2011) [1], and guides (National Chart for deontology of the professions in scientific research, 2015 [5], signed by CPU, Cirad, CNRS, Inra, Inriad, IRD, Inserm, Institut Curie).

It was presumed that theses guides and charts were unknown before the lecture, the answers to the question about requirements for quality and integrity in research, which was asked before the lecture (T0) and after the conference (T1) were expressed freely in words chosen by the respondents; the question about requirements for quality and integrity in research (Tab. 1) is therefore an open one and aimed at finding a hierarchy of items that respondents consider as parts of quality and scientific integrity. Spreading of items was therefore large and quoted items often combined. For simplification, groups of associated items are presented, for example: methodology and validated protocols, or respect, loyalty and courtesy, etc.

It appeared that the most important items are of technical nature (traceability, methodology, validated protocols) or related to behavior and management (honesty, sincerity, confidence, transparency). Together they made up for more than 40%.

During the conference, the requirements of research integrity defined in the Singapore statement were explained and therefore the understanding of this definition of research integrity and quality were surveyed after the lecture only (Fig. 3).

Figure 3 shows that a large part of the participants discovered different items which belong to the quoted charts which were previously unknown as belonging to research integrity (RI). Only 8% discovered no new items. The main items mentioned were lack of courtesy, inadequate methodology, disagreements on authorship, elimination of unpleasant measurements (e.g. “cherry picking”) and treatment of image without mention. These results indicate that for participants these items were unknown to fall under the denomination of RI, but it does not mean that participants were not aware of them being misconduct.

A general opinion about research integrity and quality was evaluated by proposing six possible answers. It appeared to be favorable as shown in Figure 4.

More than 40% of all respondents consider research integrity and quality as being “absolutely necessary” or “an evidence”. Less than 10% express that it is not easy to achieve. An equal proportion of scientists indicated that it “takes time” and that it “saves time”.

Table 1

Items that are considered part of research integrity and quality.

thumbnail Fig. 3

Extended understanding of research integrity after the conference.

thumbnail Fig. 4

General opinion on research integrity and quality.

3.2.2 Quality and integrity of personal research and that of the Unit

When asked about the frequency of scientific misconduct, most participants declared that in their opinion, scientific misconduct was not a rare phenomenon (79% when asked before the lecture, 90% when asked after). Only 15% (T0), respectively, 7% (T1) supposed that scientific misconduct was rare.

As Figure 5 shows, whether measured before or after the conference the results vary little. Few people admit that they do not, or “sometimes do not” perform good quality research and respect research integrity. A vast majority considers their own work as being conform to Research Integrity charts and of good quality, either all the time (39%, respectively 32%), or most of the time (50%, respectively 58%).

Striking at first sight, but consistent with the literature, only 3.65%, respectively, 3.8% of the scientists are confident in their colleagues' work!

thumbnail Fig. 5

Personal opinion on research integrity and quality in own work and in work of the other scientists in the lab.

3.2.3 Preferred actions to insure research integrity

In the post-lecture survey, it was asked (open question) which part of their activities the scientists would like to improve. Sixty-five percent accepted to continue this study as proposed (actions 2 and 3) (data not shown). When asked which difficulties they consider a priority, the majority (36%) of scientists opted for “traceability/data filing and architecture of filing” (Tab. 2).

The first action ended with the second survey.

Table 2

Preferred problem to work on.

3.3 Action 2: RIQ-box

The second action started a few days after the lecture and was designed to find out more about the problems with integrity and quality that scientists encounter during their work. In order to avoid any suspicion of public exposure or denouncement, it was proposed to send an e-mail message to an address, which does not identify the sender. The specific e-mail address and its use were called the “RIQ-box”. It also gave each participant the opportunity to deposit questions to be treated or observed problems with research integrity and quality. The box was used (for this study) from November 2018 to February 2019. Only six messages have been deposited (Tab. 3); no question was posed. For each observation, a possible reason, or explanation was suggested by the person reporting an item to the RIQ-box.

Table 3

Problems submitted to the RIQ-box.

3.4 Action 3: Architecture of filed data

In order to determine the efficiency of data collection and storage, several audits were performed in each laboratory. The auditor chooses a given publication on the list of publications and a table or figure arbitrarily. Auditors performed the audits according to the Inserm Chart of quality audits but did not conform to the ISO 19011 standard. In the presence of a scientist who had worked on the project, time spent to identify all intermediate and raw data used for a given table or figure was measured. They also audited whether the authors had established and architecture for easy tracing, while writing their publication “Architecture” means that raw data, treated data, averaged data etc. were filed in an organized way in order to reconstruct in a seamless way a given table, figure, photo etc. when requested.

Table 4 is a synthesis of the time spent per audit. Retrieval of raw data, treated data and additional information (batch number, experimental conditions, methodology etc.) was measured by the auditor and classified into “up to 5 min search time”, “between 15 min and 1 hr search time, “data missing” which means search time of several days or unsuccessful search. It appears that in a small majority of cases (42%), time necessary to access to all important information of a given publication, even when it was written several years ago, is very short (no more than 5 min). In most of these cases, an architecture for filing on a server and clear identification of the data had been established by the scientist. In 22 out of 28 cases, the search time was one hour or less. This search time was explained by the fact that different types of data were stored in different files or servers and no architecture of the published data had been established while writing of the publication. Sometimes several search steps were necessary to reconstruct filing.

In six cases, data (or parts of data) or necessary software to run files were missing or required more than 5 days of effort to back-track them. The comment of the auditor was that the person who had run the experiment or done analysis and synthesis had “his/her own way of filing” which was difficult to reconstruct by another person. In none of the six cases, a written architecture of data filing was available.

Table 4

Time spent to backtrack presented data to raw data.

4 Discussion

The present study is the result of a continuous engagement to ensure research quality and integrity at Inserm. Since nearly ten years, the Inserm Quality Network organizes working groups to identify possible management tools to sustain quality and integrity in biomedical research and to increase quality of life in a critical situation or a difficult environment [6]. This is the second field study conducted by the Inserm Quality Network. The first field study addressed the question of motivation and quality of life in animal housing facilities and how management can improve them [79].

It was investigated whether the scientists working in Inserm Units are sufficiently concerned with responsible research to engage in specific actions that favor impact or improve environment for quality and integrity. Eleven different Units in four different French campuses, all engaged in bio-medical research, expressed their interest within one week after a call for participation was sent out; all eleven participated in this field study. Researchers were overrepresented in the group of participants as compared to technical staff.

The study consists of three actions: the first action is a lecture on responsible research (scientific quality and integrity), preceded and followed by a survey. The head of each laboratory was present at the conference. During the second action, problems and observations were sent to an anonymous mail box. For the third action, audits of the data filing were organized and time necessary to measure the capacity of the Unit to quickly and completely find raw and treated data etc. was measured.

The aim of action 1 was to estimate the knowledge about research integrity and quality. It appears that knowledge and understanding of responsible research are insufficient. However, the interest and a favorable opinion of the importance of the matter are high. After action 1, 72% of participants wished to continue with the proposed study and perform actions 2 and 3.

These results are consistent with the importance of information and training underlined by many authors. In response to the “Rapport Corvol” from 2016 [10], which suggested, among other items, enforced information of doctoral candidates by their doctoral college (Ecole doctorale), the requirements for doctoral education in France were changed in 2016 [11]. Today, a large majority of Ph.D. students have access to specific information on research integrity and quality. Steneck [12] and other authors have pointed out the importance of education and training. Also, an increasing number of e-learning programs are available today. However, Bouter [2] pointed out that “there is no shortage of theories” about the reasons for wrongdoing in science. In addition, it should be underlined that there is no shortage of codes of conduct and guides [2,3,13]. Several authors underlined the complexity of the matter: environment can create an opportunity for misconduct, stress can be associated with diminished ethical decision making, and previous unfair treatment leads to reproduction of wrongdoing (for references see Dubois [14]).

Many authors who have investigated scientific misconduct and how to transmit best practices point out that training alone does not increase ethical decision making (e.g. see Galbraith and his references) [15].

Bouter underlines the importance of “role models” [2]. He therefore calls for “blended learning that combines online education, exercises and workgroup discussions”, in particular for “academic staff”, “supervisors and co-supervisors” and states that “an area that may be even more important than education … is quality control in the workplace”. In line with these recommendations, a training course (in French) was developed which fulfills all these requirements [4]. It is also coherent with the previous proposal of the authors to use certain tools of ISO 9001 quality management to avoid misconduct and poor methodology [3].

Scientific misconduct has alerted the scientific community since several years. Some studies of prevalence exist. They are based on the study of retracted articles reported by scientific journals of studies of the American Office of Research Integrity or on direct surveys. Each type of study holds its own specific possible bias such as probable underestimation because of lack of detection or report, or honesty of response of the respondents to surveys. In this study, for 79% (T0), respectively 90% (T1) of respondents, scientific misconduct is “not so rare”. Surprisingly, nearly all respondents had an opinion on this matter, only 6% (T0), respectively 3% (T1) did “not answer the question” or “did not know”.

In 2009, Fanelli's meta-analysis showed that 2% of scientists reported fraud (fabrication or falsification of data or plagiarism) and 34% reported questionable research practices [16]. When asked about quality and integrity of their own work, as compared to that of the members of their Unit, people made a sharp difference between themselves and the others. Ninety percent expressed that their own work respects integrity charts and of good quality (“yes” and “yes, most of the time”) whereas less than 4% said the same thing about “all members of the lab”. These findings confirm the publication of Daniele Fanelli quoted above.

The reason for this sharp difference is difficult to analyze: general defiance or the impossibility to admit one's own scientific misconduct? In previous work [9], lack of transparency, sharing of information and mutual help were used to explain lack of confidence and motivation.

Action 2 is the prolongation of the surveys. It offered the permanent possibility to report problems or observations and to ask questions. It used a specific e-mail address, which was called a RIQ-box, to which any person in the Unit could address an anonymous e-mail. The aim was (1) to gather information on what troubles the members of a Unit and (2) to make use of the gathered information to organize meetings with an Integrity manager, the Quality manager of the Unit or an invited speaker to debate and find a solution to the problems. Of course, when the person requested an immediate response he/she had to reveal a personal e-mail address to receive the answer to the question. Over the 3 months period of the study, only six observations were reported. The content of the RIQ-box revealed the entire spectrum of research misconduct (sloppy science, bad organization, cherry picking and moral misbehavior). Given the number of participating research units, the overall success of the RIQ-box was rather meager (or the frequency of observed misconduct low). Although the advantage and further use of the RIQ-box had been previously explained to the participants, whistle blowing is perceived as denunciation by many people, and its usefulness is questioned. The low number of observations may be due to the feeling of denunciation by the potential whistle blower. However, given the reported items, confidence in the other members can be considered very low, at least by those who reported to the RIQ-box, which is consistent with Figure 5. Despite the small number of observations, the content of the RIQ-box offers subjects for further discussions and more in depth information. It reveals dysfunctions in the organization, methodological problems and also research misconduct. Reported problems and often asked questions about research integrity and quality, after a course or a seminar reveal that misconduct does exist in Units and is observed by the personnel and thus supports mistrust.

Action 3 was proposed because previous quality-audits had revealed weaknesses in data filing and traceability in many laboratories. This approach is both educational and technical. When the members of a Unit discovered during an audit that treated or raw data, or certain steps of a study were unavailable or unclear, one can expect increased compliance to improve the situation. Moreover, as expected, when asked on which items the participants would preferably work, “traceability, data filing and filing architecture” gained the highest percentage (36%) of approval. In the ranking of preferences, it is followed by “methodology, validation of protocols, quality” (15%) and “training, initiation, conference” (10%). Traceability (record keeping) is not only part of Codes of conduct to responsible research, but also a specific requirement in ISO 9001 quality management. Good record keeping helps trace down decisions that had been made in the lab, agreements that were concluded and traceability of every important element of a research project. Traceability is linked to transparency, which is today a general societal request and which improves mutual trust.

A total of twenty eight figures, respectively complete articles, were audited. All of the participating Units disposed of a more or less complete Quality Management System and the audits were well accepted. Not all Units had a data filing architecture, established for the presentations of a given publication. The help of such an architecture has become quite evident. When present (and respected) it reduced search times to a few minutes. Where no architecture was used, search times were usually quite high (several hours or days). When communication within the lab is good, when personnel is rather stable and a quality manager keeps track of data and information, missing data can always be found with the help of the community. In the usual work situation in today's research, however, where the turn-over of personnel is fast and where collaborative work is the rule, a shared architecture for initial filing of data and one established during the writing of a publication, offers security and efficiency of data retrieval and is highly recommended.

5 Conclusion

This study shows that scientists (senior and junior) have insufficient knowledge of research integrity and quality. However, when informed, people adhere to the importance of research integrity and quality and understand the strong societal need for trust in research. Without training and explanation in situ, specific problems are hard to identify and to solve. Cultural influences, power of supervisors and institutions and dependence of scientists on a good reputation make some stakeholders “look the other way”. Moreover, scientists are not always aware of the help that offer simple managerial tools; therefore, ISO 9001 Quality management [17] should be given priority in the Units and used to assure integrity and quality of research.

Implications and influences

It is considered today that the impact of quality and integrity of research findings for the social acceptance of scientific research is immense and that acceptance and support of research by society is fundamental for the future of research. Without confidence in science the value of scientific results, versus opinions expressed without proof, is lost. This field study shows that awareness of scientific quality and integrity is low and not specific and that scientists express little confidence in scientific quality and integrity in their own Unit. However, once aware of the importance of quality and integrity in research, a majority is asking for help, information and training, in order to improve the reliability and traceability of results. This call should not get unanswered by institutions and in each individual Unit.

Author contribution statement

Heads of laboratory: Oumeya Adjali, Nicolas Fourcaud-Trocmé, Marc Grégoire, Régis Josien, Cyril Lafon, Patricia Lemarchand, Françoise Rédini, Philippe Saas, Christophe Tzourio, Hubert Vidal. Quality managers: Françoise Chavrier, Sabrina Chenais, Isabelle Calard, Raïssa Du Fretay, Séverine Rémy. Quality/integrity working group: Béatrice Baudouin, Edwige Pereira, Séverine Rémy.

References

  1. Singapore Statement on Research Integrity, 2011, https://wcrif.org/statement [Google Scholar]
  2. L.M. Bouter, Commentary: Perverse incentives or rotten apples? Account. Res. 22, 148–161 (2015) [CrossRef] [PubMed] [Google Scholar]
  3. E. Giesen, Efficient and ethical research management: A new challenge for an old problem, Int. J. Metrol.Qual.Eng. 6, 11 (2015) [Google Scholar]
  4. Intégrité professionnelle en recherche et innovation − Le management Ethique et Efficace (http://www.partageonslaformation.com/) [Google Scholar]
  5. Charte nationale de déontologie des métiers de la recherche (National Chart for deontology of the professions in scientific research) 2015, https://www.cnrs.fr/comets/IMG/pdf/charte_nationale__deontologie_signe_e_janvier2015.pdf [Google Scholar]
  6. R. Bareille, B. Baudouin-Massot, M. Carreno, S. Fournier, N. Lebret, I. Remy-Jouet, E. Giesen, Preventive actions to avoid questionable research practices. Use of Ethical and Efficient Research Management during arrival and departure of a co-worker, Int. J. Metrol. Qual. Eng. 8, 10 (2017) [CrossRef] [EDP Sciences] [Google Scholar]
  7. J.D. Arnaud, V. Chauffeton, V. Gall, Y. Lottin, A.M. Mura, E. Giesen, Manager un service de zootechnie: quelques conseils pour favoriser la qualité de vie au travail, STAL 42, 1–9 (2016) [Google Scholar]
  8. A.M. Mura, Y. Lottin, J.D. Arnaud, V. Chauffeton, V. Gall, E. Giesen, L'animalerie prise en tenaille, STAL 42, 27–36 (2016) [Google Scholar]
  9. Y. Lottin, Y. Barreira, T. Chassat, V. Chauffeton, V. Corcelle, C. Evra, A. Le Bon, K. Mesbah, A.M. Mura, S. Pellegrino, H. Pointu, B. Rault, P. Rostagno, A. Savarous, M.F. Tinland, E. Giesen, Manquons-nous de qualité de vie en zootechnie?, STAL 40, 17–37 (2017) [Google Scholar]
  10. Bilan et propositions de mise en œuvre de la charte nationale d'intégrité scientifique, http://cache.media.enseignementsup-recherche.gouv.fr/file/Actus/84/2/Rapport_Corvol_29-06-2016_601842.pdf [Google Scholar]
  11. (JORF n°0122 du 27 mai 2016, texte n° 10, Arrêté du 25 mai 2016 fixant le cadre national de la formation et les modalités conduisant à la délivrance du diplôme national de doctorat. NOR: MENS1611139A. https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000032587086&categorieLien=id [Google Scholar]
  12. N. Steneck, Global research integrity training, Science 340, 552–553 (2013) [Google Scholar]
  13. NF X 50-553 Management des activités de recherche (2014) [Google Scholar]
  14. J.M. Dubois, E.E. Anderson, J. Chibnall, K. Carrol, T. Gibb, C. Ogbuka, T. Rubelke, Understanding research misconduct: A comparative analysis of 120 cases of professional wrongdoing, Account Res. 20, 320–338 (2013) [CrossRef] [PubMed] [Google Scholar]
  15. K.L. Galbraith, Life after research misconduct: Punishment and the pursuit of second chances, J. Empir. Res. Hum. Res. Ethics 12, 26–32 (2017) [Google Scholar]
  16. D. Fanelli, How many scientists fabricate or falsify research? A systematic review and meta-analysis of survey data, PLoS ONE 4, e5738 (2009) [CrossRef] [PubMed] [Google Scholar]
  17. ISO 9001: 2015 ISO 9001 Quality management systems − Requirements [Google Scholar]

Cite this article as: Monique Estienne, Catherine Chevalier, Catherine Fagard, Patricia Letondal, Eva Giesen, Responsible scientific research at Inserm: a field study, Int. J. Metrol. Qual. Eng. 11, 1 (2020)

All Tables

Table 1

Items that are considered part of research integrity and quality.

Table 2

Preferred problem to work on.

Table 3

Problems submitted to the RIQ-box.

Table 4

Time spent to backtrack presented data to raw data.

All Figures

thumbnail Fig. 1

Participants in the study.

In the text
thumbnail Fig. 2

(a) Proportion of trained/untrained scientists. (b) Prevalence of short training in research integrity

In the text
thumbnail Fig. 3

Extended understanding of research integrity after the conference.

In the text
thumbnail Fig. 4

General opinion on research integrity and quality.

In the text
thumbnail Fig. 5

Personal opinion on research integrity and quality in own work and in work of the other scientists in the lab.

In the text

Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.

Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.

Initial download of the metrics may take a while.