We, as a lab, value and strive to advance the mission of open
science to improve the accessibility, reproducibility, and
replicability of science. As such, all lab members are expected to
conduct research transparently and to promote reproducibility. This
includes, but is not limited to, pre- or
co-registering studies, sharing analysis scripts
and data, using version control
(GitLab), submitting preprints when
submitting a manuscript to a journal, and providing support for other
labs’ attempts to replicate and reproduce our findings. Our lab’s
template for projects on the Open Science
Framework (OSF) is located here1: https://osf.io/4w9sv.
2 Pre-Registration (or
Co-Registration)
There is a continuum of registration approaches. Pre-registration
involves publicly posting aspects of a study (e.g., study design,
hypotheses, methods, materials, and analysis plan) before data
collection begins. Co-registration involves specifying aspects of your
study after data collection starts but before analysis.
Post-registration involves specifying aspects of your study after after
analysis has begun.
Pre-registration is often considered the gold-standard. However,
co-registration and post-registration are better than no
registration.
Here are elements of a study that can be registered:
study design
hypotheses
methods
materials
analysis plan
When analysis decisions are contingent upon prior steps that may
influence which decision to take, the analysis plan can include decision
trees (e.g., if X, then Y; if A, then B). For example, you can specify
how you would proceed if the measures do not demonstrate longitudinal
factorial invariance, or how you would handle poorly fitting models.
3 Sharing Data, Analysis
Scripts, and Research Materials
3.1 Data Dictionary
For each study, we create a Data Dictionary. A Data Dictionary is a
metadata file that tells people the meaning of variables in the data
file and how to interpret them.
3.1.1 Style Guide
Use Roboto font size 10
Use an en dash (–; i.e., not a hyphen) to indicate a
range:
e.g., 1–18 (not 1-18)
an en dash is technically correct; in addition, spreadsheets often
read 3-7 as March 7th, but they correctly read 3–7
3.1.2 Columns
The Data Dictionary should have the following columns:
Variable Name
the variable name in the data file
Form Name
the instrument or measure that the variable comes from
Human-Readable Variable Name
a more easily readable version of the variable name
Data Type
the format of the values in the column (e.g., string, binary,
integer, numeric, date, time, etc.)
Variable Type
whether the scale of measurement is nominal, ordinal, interval, or
ratio
Measurement Unit
the conceptual unit that is being measured (e.g., seconds, level,
count)
Allowed Values
the allowed values for a variable and (if possible), what conceptual
level each value corresponds to (e.g., 0 = Male; 1 = Female)
Description
conceptual description of the variable
Definition
definitions of abbreviations, conceptual defintions of terms,
mathematical definitions of how a variable is calculated, etc.
Notes
additional notes about the variable and how it is calculated
References
references for the measure and/or variable
3.1.2.1 Data Types
Data Types include:
string
include letters or other characters (and possibly numbers)
factor
categorical variable with letters and/or numbers
binary
0/1
integer
whole numbers (never decimals)
numeric
numbers
date
MM/DD/YYYY; e.g., 06/24/2020
time
HH:MM:SS (e.g., 01:30:24), or HH:MM (e.g.,
05:24), or MM:SS
date-time
MM/DD/YYYY HH:MM:SS (e.g., 06/24/2020 01:30:24)
3.1.2.2 Variable
Types
Variable Types include:
nominal
distinct categories
ordinal
ordered categories
interval
ordered with meaningful distances
ratio
ordered with meaningful distances and an absolute zero
3.1.2.3 Measurement
Units
Measurement Units include:
ID
participant identification (ID) numbers
count
number of something (e.g., number of children in the household)
group
categories that reflect different groups (e.g., female vs male)
instance
nominal categories that do not reflect groups
yes/no
0 = No; 1 = Yes
ratio
ratio of two variables (one variable divided by another
variable)
USD
U.S. dollars ($)
option
categories that reflect participant’s choice among multiple
options
location
categories that reflect different locations
state
categories that reflect a status
degree
the degree of
level
the level of
grade
school grade
date
MM/DD/YYYY; e.g., 06/24/2020
time
HH:MM:SS (e.g., 01:30:24), or HH:MM (e.g.,
05:24), or MM:SS
When submitting a manuscript to a journal, also submit a preprint to
PsyArXiv. Combine the supplemental
material and manuscript into one PDF file when posting.
6 OSF
The Open Science Framework (OSF) is a
website for hosting pre-registrations, data, analysis code,
research-related materials, and pre-prints to improve replicability and
reproducibility of findings. For each paper project, create a new
repository on the OSF and add the relevant contributors, including
Dr. Petersen. Our lab’s template for projects on the Open Science
Framework (OSF) can be found here3: https://osf.io/4w9sv.
6.1 Components
Create a component in the OSF project repository for the following
components:
Pre-registration (or Co-registration)
Data
Data Dictionary
Analysis Code
Research Materials
Preprint
6.1.1
Pre-registration
6.1.2 Data
To help protect participant anonymity, it is important to anonymize
participant IDs so their data cannot be stitched together across papers.
To anonymize participant IDs, use the following script and change the
seed for every paper so that a given participant gets a different
anonymized code each time.
For each paper project, we export a .csv file with the
subset of the Data Dictionary variables
used for that specific paper. We upload that .csv file to
the OSF. The formatting of the Data Dictionary is described here.
6.1.4 Analysis Code
6.1.5 Research
Materials
6.1.6 Preprint
6.2 Create Anonymous
View-Only Links for Anonymous Peer Review
Before submitting a manuscript to a journal, make sure to post the
relevant materials on the OSF, as described here, and post the preprint, as described here. When preparing a manuscript for submission to
a journal, make sure to follow the Author Guidelines for each journal.
After finalizing the manuscript in accordance with journal guidelines
and when you are ready to submit the paper to the journal (but before
submission), post the preprint on PsyArXiv. Include the link to the
preprint in the cover letter to the journal. In the method section and
on the title page, include the relevant OSF links to the
pre-registration, data, data dictionary, analysis code, computational
notebook, and research materials, etc. For example:
Hypotheses and measures for the School Readiness Study were
pre-registered: https://osf.io/jzxb8. Hypotheses methods, and a data
analysis plan for the present study were also pre-registered: https://osf.io/pny26. Data
files, a data dictionary, analysis scripts, and a computational notebook
for the present study are published online: https://osf.io/zs2bn.
In the manuscript submission, create and use anonymous view-only OSF links (for
blind review). In the preprint submission, use the general OSF links
(not the anonymous view-only OSF
links) that will become viewable when the manuscript is accepted for
publication (i.e., when you make the OSF repo public).
8 When the Manuscript is
Accepted for Publication
When the manuscript is accepted for publication:
Let all of the authors know, and send them the full, (in-press)
APA-style reference
Make the OSF repo public and create a DOI link
for the repository that can be used for citing it
Submit the finalized, unblinded manuscript (and any tables, figures,
and the supplement; with the public OSF link; removing any highlighting
or tracked changes) to the NIHMS system: https://www.nihms.nih.gov/submission/create/
After you submit the manuscript to NIHMS, send Dr. Petersen the
NIHMS ID for the submission. We are required to report published papers
to funding agences.
Make sure the finalized, unblinded manuscript (and any tables,
figures, and the supplement; with the public OSF link; removing any
highlighting or tracked changes) is uploaded as an updated version of
the preprint on PsyArXiv.
