On November 26, 2019, the US Office of Science and Technology Policy (OSTP) issued a “Request for Information on the American Research Environment.” The Association of Research Libraries (ARL) welcomes this opportunity to suggest ways that US federal agencies, in collaboration with the research community, can improve the research environment.
Association of Research Libraries Response to US Office of Science and Technology Policy (OSTP) Request for Information on the American Research Environment
January 28, 2020
Thank you for the opportunity to provide input on the actions that US federal agencies, in collaboration with the research community, can make to enhance the American research environment. I submit the following views on behalf of the Association of Research Libraries (ARL), a nonprofit collective of libraries in 124 leading research institutions in the United States and Canada. Of the four main topics outlined in OSTP’s request for information, the research library community is most active in the areas of “Research Rigor and Integrity” and “Coordinating Administrative Requirements for Research,” to which these comments are targeted. As collaborative partners supporting the full life cycle of scientific inquiry and creation, our mission is to create an equitable, enduring, and barrier-free research information environment to advance research and learning. ARL member libraries contribute to research rigor and integrity by providing expertise and education on open science tools, practices, and ethical data sharing, by partnering with research administrative offices on ensuring compliance with federal requirements, and ultimately in providing repository and data curation services for current and future scholars. We offer the following recommendations for US federal agencies:
Open science practices improve research rigor and integrity
ARL endorses the recommendations in the 2018 National Academies of Science, Engineering, and Medicine (NASEM) consensus report Open Science by Design: Realizing a Vision for 21st Century Research. The report, grounded in FAIR principles, promotes essential actions for research ecosystem stakeholders to improve openness and transparency in research processes, and share and reuse research products, in order to accelerate scientific discovery and innovation.
In particular, research funders and research institutions are in the best position to develop policies and procedures to identify the data, code, specimens, and other research products that ensure long-term public availability, and they are best positioned to provide the resources necessary for the long-term preservation and stewardship of those research products.1 Successful implementation of policies to identify research outputs for reuse and long-term preservation will require integration and alignment between the scientific community (e.g., managers of domain repositories and scholarly societies) and the stewardship community. ARL is committed to partnering with and convening the relevant stakeholders to work towards this alignment.
Require active, machine-readable data management plans
ARL recommends that federal agencies require machine-readable data management plans (DMPs) to facilitate better communication between researchers and their institutional research support units (including the library) and in reporting progress to their funders. At a National Science Foundation–funded conference on effective data practices in December 2019, stakeholders including university research officers, scientists, and librarians agreed that data management planning is important for sharing and use of research data and outputs. Participants suggested that the ability to update plans (“just in time”) across the project life cycle and as part of progress reporting would accelerate the value and adoption of DMPs among researchers, beyond what is required for compliance.
Develop coordinated approaches to public access policies
ARL is working with the Association of American Universities (AAU) and the Association of Public and Land-grant Universities (APLU) to harmonize intra- and inter-institutional processes related to data management planning and data sharing. ARL endorses the November 2017 “AAU-APLU Public Access Working Group Report and Recommendations,” including recommendations for federal agencies to:
- strive to harmonize their policies, documentation (e.g., data management plan, data use agreement terms, and data sharing certifications), and compliance approach to public data access to the greatest extent possible to minimize the administrative requirements and costs for both agencies and funding recipients.
- continue to support universities’ retention and free accessibility of data obtained with federal funding, in analogy to the treatment of IP. Third-party vendors who may be contracted by universities and other funding recipients to assist with data access should not be granted an exclusive license to such data or other exclusive rights to make such data available in the future.
- recognize that before an institution publicly shares data, it must be evaluated for quality and well-documented to ensure the data adds value to the scientific discussion, can be understood and effectively reused, and can withstand public scrutiny.
- align policies with standards and guidance on retention, security, and data use terms developed by the research community.
- clarify and continue to explicitly note in their calls for proposals that costs to support a program’s requirements for data accessibility are allowable as direct or indirect charges in research program budgets.
- emphasize adherence to the broadly accepted FAIR principles (data should be findable, accessible, interoperable, and reusable) in data access rules.2
Promote or require study pre-registration and protocol repositories
ARL recommends that federal agencies expand the promotion and requirement of study pre-registration, including as a required part of the merit review process for funding, where discipline-appropriate. Pre-registration of a research study design, before data is collected, is a practice that (1) expands the discoverability of research because it includes unpublished studies in the scholarly record, and (2) addresses potential biases that interpretation of data can have on a published study’s stated objectives. Pre-registration is required in clinical trials and has been successful in addressing both of these concerns.
Similarly, protocol registrations allow researchers to publish research procedures and methods, often in a machine-readable format. Practices such as pre-registration and protocol registration are critical aspects of reporting that facilitate reuse and replication of research, and reduce inefficiencies in the research ecosystem. A recent NASEM consensus report recommends and supports the use of these mechanisms to improve the rigor and reproducibility of research.3
Encourage publication of negative findings or outcomes
ARL recommends that federal agencies encourage the publication of negative or null findings and reward such publications in the merit review process. Publication of negative findings will increase the transparency of the research process, reduce bias in the scholarly record, and accelerate the progress of science. Publishing null results will also reduce the waste associated with duplicating experiments without knowledge of prior work.
One solution in the open science community for sustainably expanding publishing output (necessary to include null findings) is through preprinting. The Center for Open Science, which hosts a number of preprint services on its platform, suggests, for example, that “ when adopted at scale, preprinting can produce a virtuous system in which all content is openly available, and evaluation services (journal editors and reviewers) focus on improving the quality of that content and the visibility of relevant content to readers. Evaluation services help readers decide what to read rather than controlling their ability to read it.”4
Provide funding for replication studies and metascience
ARL recommends that US federal agencies fund replication studies, multiple investigators or labs to work on the same question, and meta-scholarship based on existing publications and data. Replication studies, such as those coordinated by the Center for Open Science in psychology and cancer biology, have demonstrated the value of such studies in validating findings and methods, and improving the processes and documentation associated with making work replicable. Such lessons are critical for improving education and training of scientists, as well as tool developers who can help automate documentation and reporting. ARL further recommends that agencies fund meta-research to assess bodies of research for rigor and integrity, as well as to combine results from multiple studies to assess an overall effect.
Addressing the tension between funding new science and funding replication, the National Academy of Sciences 2019 report Reproducibility and Replicability in Science contains proposed eligibility criteria for replication studies that ARL supports.
Provide funding for curation
ARL recommends that research funding include allowable costs for data curation, including metadata. Curation—which includes activities and treatments taken across the published data life cycle to ensure discoverability and its ongoing preservation—is necessary to ensure availability for reuse. Research librarians work with scholars and scientific domain experts to create structured metadata and documentation, and encourage the adoption of community standards to make research outputs machine actionable and AI-ready.
Support new models of open infrastructure and publication
ARL recommends that federal agencies provide maintenance funding and require maintenance plans for community-governed tools and services that enable rapid dissemination, interlinking research through registries of persistent identifiers, data sharing, and collaboration to advance scientific progress. New modes of research publication enable researchers to publish executable code and data alongside articles, share preprints with associated data and code, enable post-publication peer review through overlay journals, and facilitate collaboration and team science.
Scientific tools and infrastructure such as outlined above, including tools like Jupyter Notebooks, ReproZip, and Code Ocean, accelerate the progress of science and facilitate replicability. Openness enables both interoperability and preservation for future research and the scholarly record. A recent paper on the arXiv.org preprint server, “Publishing Computational Research—A Review of Infrastructures for Reproducible and Transparent Scholarly Communication,” provides an excellent review of the issues from major stakeholder perspectives.5
Support expanded education and training
ARL recommends that federal agencies provide funding for training in open science practices, including those that promote replicability and reproducibility, good data management, methods to identify and mitigate against bias in experimental design, and research ethics, particularly as they relate to AI. ARL member libraries work with other units on campus and independently to provide training to graduate students, postdocs, and faculty. Funding to further develop cross-unit skills in providing training would further advance research integrity in current and future generations of scholars.
Thank you for your consideration of these comments.
Mary Lee Kennedy
Association of Research Libraries
- National Academies of Sciences, Engineering, and Medicine, Open Science by Design: Realizing a Vision for 21st Century Research (Washington, DC: National Academies Press, 2018), 10, https://doi.org/10.17226/25116.
- Association of American Universities and Association of Public and Land-grant Universities, “AAU-APLU Public Access Working Group Report and Recommendations,” November 29, 2017, 2–3, https://www.aau.edu/sites/default/files/AAU-Files/Key-Issues/Intellectual-Property/Public-Open-Access/AAU-APLU-Public-Access-Working-Group-Report.pdf.
- National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science (Washington, DC: National Academies Press, 2019), https://doi.org/10.17226/25303.
- David Mellor, Brian Nosek, and Nicole Pfeiffer, “Conflict between Open Access and Open Science: APCs Are a Key Part of the Problem, Preprints Are a Key Part of the Solution,” Center for Open Science Blog , January 21, 2020, https://cos.io/blog/conflict-between-open-access-and-open-science-apcs-are-key-part-problem-preprints-are-key-part-solution/.
- Markus Konkol, Daniel Nüst, and Laura Goulier, “Publishing Computational Research—A Review of Infrastructures for Reproducible and Transparent Scholarly Communication,” preprint, submitted January 2, 2020, https://arxiv.org/abs/2001.00484.
About the Association of Research Libraries
The Association of Research Libraries (ARL) is a nonprofit organization of 124 research libraries in Canada and the US whose mission is to advance research, learning, and scholarly communication. The Association fosters the open exchange of ideas and expertise, promotes equity and diversity, and pursues advocacy and public policy efforts that reflect the values of the library, scholarly, and higher education communities. ARL forges partnerships and catalyzes the collective efforts of research libraries to enable knowledge creation and to achieve enduring and barrier-free access to information. ARL is on the web at ARL.org.