On April 18 and 19, the Brazilian Reproducibility Network held an online conference on Reproducibility in Brazilian Research, as a preparatory event for the 5th National Conference on Science and Technology. There were two days of debates on the meaning of reproducibility in the natural sciences, health sciences and humanities, as well as discussions on the roles that institutions, funders and journals can take on to promote reproducible research practices. The recording of the two-day event is available on Youtube (day 1 and day 2) and the conference summary is available here.
During the conference, CAPES' evaluation director, Dr. Antônio Gomes de Souza Filho, invited the BRN to draw up a list of practical recommendations for incorporating open and reproducible research practices into the evaluation of Brazilian postgraduate courses. The drafting of the document was led by the BRN, with extensive collaboration from members of the Open Government Partnership (Commitment 3) and our community. The document is available here and contains 8 recommendations, based on international initiatives to reform scientific evaluation and adapted to the context of postgraduate evaluation, which are summarized in the figure below.
CAPES' evaluation takes place every four years, validating the country's master's and doctoral programs and informing the direction of the federal government's investments in the National Postgraduate System. As such, the evaluation has great potential to direct research practices within the programs. In recent evaluations, the assessment criteria related to the quality of intellectual production have been strongly linked to the Qualis ranking, which in turn is based on other bibliometric indicators of journals, such as the impact factor.
Various international initiatives such as DORA, the Leiden Manifesto, CoARA and the Hong Kong Principles recommend that the publication vehicle should not be used to assess the quality of research, which is the first of our recommendations. We also advocate for the qualitative evaluation of selected output, directed towards different quality dimensions such as methodological rigor and transparency.
We also recommend that the open availability of articles, theses and dissertations be considered, as well as the sharing of data, materials, codes and other research products. How open access is achieved should be up to researchers, and should not depend on the payment of publication fees in the case of articles.
Other recommendations revolve around the training and support offered to students to adopt open and reproducible research practices. Graduate programs should include discussions on these topics in the curriculum, as well as implement policies and support services in this regard. Open education practices that make courses and subjects offered by the program available to a wider audience should also be valued.
Finally, we recommend that the assessment of the research output of researchers and students should include dimensions other articles and theses. Activities such as pre- and post-publication peer review, editing scientific journals, participating in organizations and committees dedicated to scientific policies, and taking up roles in stable research collaborations should also be taken into account.
The recommendations were presented by the BRN to the Technical-Scientific Council for Higher Education on May 21, and we are open to developing recommendations in this regard for other agencies, institutions and graduate programs.
If you are interested in adapting our recommendations to your context, please access the document and contact us via redereprodutibilidade@gmail.com.
Comments