Best Practices for Running a Hyperfunctional Psychology...
Transcript of Best Practices for Running a Hyperfunctional Psychology...
Best Practices for Running a
Hyperfunctional Psychology
Laboratory
Greg J. Siegle, Ph.D.
University of Pittsburgh
School of Medicine
Presented work supported by MH082998
These slides available at
http://www.pitt.edu/~gsiegle/SiegleLaboratoryBestPracticesColloquium.pdf
Why bother? • You and others can trust your data
– It’s easy to know when you step into a best-practices lab
– Some researchers get a reputation as “careful”
• Increase replicability
• Decrease debacles
– Example from my lab:
• The chilling chiller incident
– Example from the current fMRI world
Pre-emptive strike:
Clinical Operations Manual
INCLUDING template
documents • Common elements
– Study Responsibility Log – who does what when
– Study worksheet – stuff which has to happen and when, e.g., calibrations, audits
– Assessment Schedule
– Assessment Grid
– Procedural Checklists
– Regulatory Binder Template From http://www.uth.tmc.edu/ctrc/studymanagement.html
Regulatory Binders & Lab documents • Basic clinical trial model
– 2 folders per patient – 1 for identifiable info, 1 for all study documents. + master list.
• Excellent list of lab documents – http://www.uth.tmc.edu/ctrc/regulatory.html
– Binders/Folders for • Protocol and amendments
• Data – Subject Logs and Lists
– Patient Data – 1 per participant
– Contact Logs and monitoring
• Reporting – Corrospondence with outside organizations (e.g., FDA)
– IRB Documents
– Case Report Blank Forms
– Adverse events
• People – Investigator Information
– Team Information
• Lab information – Lab certifications, etc.
– Equipment
– Investigational product (e.g., drug) info
• Meeting documents – Study meetings
– Study reports
• Publications
Study management database
Stuff to include in addition to data
• Subject information – Screening/Enrollment log
– Visit Schedule Log
• Tracking/Reporting information – Adverse Event Log
– Protocol Deviation Log
– Data Cleaning log
• Accountability logs – Device calibrations and accountabilities
• Note: SEPARATE database for Master subject log
Quality management plan
• what
will you
check,
how
will you
check
it?
Data collection Data processing
Create folders
• Study folders: at least – data
• pupil
• heart
• behav
• ….
– analysis • matlab
• spss
– documents
– publications
– regulatory
– software
Folder contents (from Dr. Nicole Prause) • Data
– Raw, important processed stages, data processing scripts such as .m file backup, compiled data, final data
– The data folder should contain enough information to quickly reconstruct important phases of data processing without storing too many large files on the computer indefinitely.
– Every data folder should include is a "notes.txt" file, where you note abnormalities for particular subjects and files to enable quick reconstruction of data sets. For example, if a person becomes ill and withdraws from the study, it will be much easier to find this noted in a single file than to start searching to understand why the last two test conditions are missing to make decisions about data inclusion/exclusion.
• Institutional Review Board Compliance – Submissions, revisions, letters of approval, up-to-date informed consent
• Scripts – Electronic questionnaires, up-to-date DMDX scripts, backup of stimuli if size
reasonable
• Publication – Poster presentations, papers being prepared, final drafts of accepted/published
papers
Select protocols
carefully • Stay as close as possible to
industry standards when possible (deviating as necessary…) – E.g., the Society for
Psychophysiological Research has published standards for EEG, ERP, Startle, Heart rate, HRV, EMG, disease transmission… • http://www.sprweb.org/journa
l/index.cfm
– Internet questionnaires: Skitka • www.uvm.edu/~pdodds/files/
papers/others/2006/skitka2006a.pdf
– ASTM (standards body) • www.astm.org
Procedural checklists & records
• Every detail is golden: Have checklists and how-to guides
• Check the checklists – “All records shall be prepared, dated, and
signed (full signature, hand written) by one person and independently checked, dated, and signed by a 2nd person” (GMP (Good Manufacturing practices) 211.186)
• Electronic checklists? – Possible
• “Electronic records may be considered trustworthy and reliable and be used in leiu of paper records provided that the electronic records have proper secuirty controls” (21 CFR Part 11 Subpart A Sec 11.1)
• “Ensure authenticity & integrity of electronic records such that the person responsible for the electronic record cannot readily repudiate the record as not genuine” (21 CFR Part 11 Subpart B Sec 11.10)
• Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a))
– But I don’t recommend it yet!
Video – more is better
• Essential for clinical interviews to at
least get audio. Video is better.
• Note: Need IRB Approval
Task design
• Validation
– Check timing / event logging
• w/ fMRI we test at the scanner 1x phantom + 1x pilot before any protocol
– Check single subjects
• Write analysis scripts for single subjects BEFORE your first real subject
• Be a subject for your own protocols
• Test everything completely BEFORE your first pilot subject.
• Test everything completely BEFORE your first real subject.
Psychophys lab setup
• Neat reproducable lab setups – Diagram in your Ops Manual
to show how to do stuff exactly the same every time
– As many procedural diagrams as might be useful
• Care about disease transmission – Bloodborne Pathogen control:
• Gloves – as much as possible
• Don’t abraid the skin more than you need to
• Disposable electrodes when possible
• Disinfect – CIDEX if you have ventillation
– Control III + Suave shampoo if you don’t
• Wear a labcoat – that’s actually what they’re for
Dr. Nicole Prause’s lab setup
http://www.span-lab.com/Assets/images/photos/EEGprep.JPG
Checking stuff works before data
collection
• Protocols before your protocols
– Check all communications between computers, peripherals, and data collection devices
– Make sure your stimuli show
• Have this in your checklists
• We have eprime routines to test
– getting scanner trigger,
– eye-tracker events
– mouse/button pushes
Data Security & Integrity
• Whitebox standards: – Keep original data in unalterable form
– Have 2nd copy for any necessary changes (e.g., remove a few trials, concatenate runs…)
– Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a))
• Security – 21 CFR part 11:
• Double password protection
• Standards – They exist for most things: http://www.astm.org/
• IRB – E.g., consent forms separate from data
Databases
• Huge science - http://c2.com/cgi/wiki?DatabaseBestPractices
• E.g., – Have primary keys
– Don’t change schemas
– Consistent long descriptive column names across tables
– Try things first in a local database
– Good rule of thumb: 20 columns per table – more is weird design
• Lab standards – Ids are in columns called “id”
– All tables have id
• 21 CFR Part 11 – Keep an audit history of date created and by who, and dates
changed/updated
Backups
• Ideally
– Daily data backups
– Weekly incremental computer backups
– Monthly full backups
• Keep a set of backups in a secure place outside
your lab
Documentation • Document everything
• Lab notebooks are essential – Extreme: Open Lab Notebook
• http://en.wikipedia.org/wiki/Open_Notebook_Science
• All work posted immediately to the public eye
• Good tool: http://openwetware.org/wiki/Main_Page
– Commercial approaches • Big list at:
http://campusguides.lib.utah.edu/content.php?pid=126157&sid=2131670
– My approach: Powerpoints per study • Greg’s Journal template – on the
PICAN server – \\oacres3\rcn\pican\docs\gjsjourna
l.pot
– Sharepoint blog?
– Database page for all changes with name, date, change description
• Analyses should be reproducible – I like 1 matlab or SPSS file with
all commands that produce all analyses for a given study.
Reasons for using ELNs/
virtual workspaces
• 1. They are an efficient way of managing large projects, multiple
projects and multi-institution projects.
• 2. Provenance ensures that any accusation of fraud can easily be addressed.
• 3. Addresses the problem of missing information due to turnover in lab personnel (and students).
• 4. Can access research results from anywhere and therefore keep up with the ongoing work in the lab while traveling.
• 5. These systems are already being used in industry, therefore are studentsneed to be acquainted with them to be employable.
• 6. Meets requirements of granting agency mandates for data managment plans.
• 7. Facilitates depositing data into data repositories for reuse and repurposing.
http://campusguides.lib.utah.edu/content.php?pid=126157&sid=2131670
Beyond Powerpoint
Lab Bench People layer
http://campusguides.lib.utah.edu/content.php?p
id=126157&sid=2131670
Example commercial solution:
(Not endorsed just summarized) • From labarchives.com
– Intuitive Electronic Lab Notebook (ELN) organizes your laboratory data
– Preserve all your data securely, including all versions of all files
– Share information within your laboratory
– Keep abreast of developments in your lab even when traveling
– Collaborate with investigators by sharing selected data from your Electronic Laboratory Notebook
– Publish selected data to specific individuals or the public
– Protect your intellectual property
– Runs on all platforms, including Windows, Mac, Linux, iPad and Android devices Special classroom version of our Electronic Lab Notebook also available
Sample all-figures-in-paper script
%% associations of power change with change in other things within and between groups
ctrl=find((s.grp==1) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));
cct=find((s.grp==2) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));
tau=find((s.grp==3) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));
cct_tau=[cct; tau];
fprintf('----------------------------------\n');
fprintf('CCT r(power_on change, rumination change)\n');
st.r_powerOnChg_rsqchg_CCT=r(poweronchg(cct),s.rsqchg(cct),0,1,1.5,-999);
figure(7); clf;
regplot(rescaleoutliers(poweronchg(cct)),rescaleoutliers(s.rsqchg(cct)));
xlabel('Trial Frequency Power Post CCT - Pre CCT');
ylabel('Rumination (RSQ) Post CCT - Pre CCT');
figure(8); clf;
regplot(rescaleoutliers(poweronchg(tau)),rescaleoutliers(s.rsqchg(tau)));
xlabel('Trial Frequency Power Post TAU - Pre TAU');
ylabel('Rumination (RSQ) Post TAU - Pre TAU');
figure(9); clf;
focindfromcct_change=-9.94-151.94.*poweronchg+109.13.*poweroffchg;
regplot(rescaleoutliers(focindfromcct_change(cct)),rescaleoutliers(s.rsqchg(cct)));
xlabel('Unfocus Index Post CCT - Pre CCT');
ylabel('Rumination (RSQ) Post CCT - Pre CCT');
Use best practices for
preprocessing data
• Again with the Psychophysiology guidelines • http://www.sprweb.org/journal/index.cfm
• Visual inspection of artifacts
• When are artifacts ok to let in to data?
– How much should we say we’re letting in?
• Contingency planning
– What if you change preprocessing midway through?
• I think you should reprocess everything
– What if you change preprocessing after-the-fact?
• Depending on how serious, note it.
Quality control
• Diagnosis and Clinical
dispositions:
– Case conferences
• Reliability on
ANYTHING subjective
• Double data entry
• See your “Research
Methods” textbook…
Check your data early and often
• Quality check psychophys data that day and
fMRI data within a week (while it’s on the
servers)
• Single subject analyses
• Group analyses with N=5
Calibrations
• Regular – monthly calibrations of all
instruments
– Currently done for pupilometer
– Other stuff?
• MR center and BIRC have done calibrations,
e.g., stability checks regularly. We don’t
request them. But we should for our own
documentation.
Security
• Double-locked file cabinets
• Password protection for computers, files, etc.
• Note: 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records
– has security standards for data
– audits, system validations, audit trails, electronic signatures, and documentation
– http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=11
Audits
• Every 6 months, all data within that 6 months
• Quality management help at: – http://www.uthouston.edu/CT
RC/trial_conduct/quality-management.htm
– There are chart audit tools
– Regulatory file review tools
• Every year – full audit – should be easy
The human thing
• Laboratory mentality is important. Attend to it.
Anecdotal evidence suggests happy inspired
labs are often more functional.
• You will likely not be in touch with the
emotional health of the lab. Have someone
who is. Make their report to you on lab health
a regular thing.
Hire for your weaknesses
• Good labs often have people who are (not all
of these at once)
– Detail oriented
– Socially attuned
– Tech savvy
– Inspired
Sources
• 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records – has security standards for data
– audits, system validations, audit trails, electronic signatures, and documentation
– http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=11
– Esp. Subpart B – electronic records
• Good Manufacturing Practices (GMP)
• ASTM (standards body) – www.astm.org
– Robert L. Zimmerman Jr, 10 Best Practices for Good Laboratories. Nov Dec, 2010, November/December, Standardization News
• Clinical Trials Resource Center – http://www.uthouston.edu/CTRC/trial_conduct/