There’s No Avoiding It: Programming Skills You’ll Need
-
Upload
yannick-pouliot -
Category
Health & Medicine
-
view
70 -
download
0
description
Transcript of There’s No Avoiding It: Programming Skills You’ll Need
![Page 1: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/1.jpg)
Yannick Pouliot, PhD 10/14/2011
There’s No Avoiding It: Programming Skills You’ll Need
![Page 2: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/2.jpg)
Three Things I want To Impress
•Why software programming is essential for bioresearch▫… as essential as knowing how to use a
pipette•Why you should partially dump Excel and
use a relational database•Why the Cloud is your friend
![Page 3: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/3.jpg)
•Free software!•Free algorithms!•Pre-coded algorithms (i.e., packages)!•Very cheap computing power!
The Good News
![Page 4: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/4.jpg)
The Bad News
•Dunno how to use•“Not talented”•“Not enough time”•(can’t be bothered)
▫e.g., reading the paper describing the software tool one is relying on
![Page 5: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/5.jpg)
More Good News
•Not that hard•Lots and lots of good resources•Read a book, dammit•Find a buddy•Use Cloud instances (preconfigured
machines)▫Can even be free!
![Page 6: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/6.jpg)
The Quest For Situation-Appropriate Storage & Computation
Or, when Excel fails you
![Page 7: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/7.jpg)
Some Questions…
1. Do you use MS Excel?2. How much time do you spend using it?3. Are you good at it? Be honest…4. Have you ever read a book or tutorial on
Excel?5. So how are you going to improve your
ability?
![Page 8: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/8.jpg)
Are You an Excelaholic?
•Do you have an unhealthy dependence on Excel?▫Do you use Excel to store data?▫Do you feel like you’re making Excel jump
through hoops to perform your calculations? Do you have a vague feeling of shame as a
result?
![Page 9: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/9.jpg)
The Worst Case (More Frequent Than You’d Wish)
•Postdoc uses Excel to keep track of complex experiment involving two external groups
•Eventually realizes that data stored in Excel were corrupted (“paste failure”)▫Result: it took her six months to recover
•She now uses FileMaker (relational database)
![Page 10: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/10.jpg)
The Next Level Up: Relational Databases Take Your Pick
![Page 11: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/11.jpg)
A Real Example From Yours Truly
![Page 12: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/12.jpg)
But You Also Need Programming…
![Page 13: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/13.jpg)
Why Programming?
•Address small problems that can nail you•Address bigger problems by standing on
the shoulders of giants•Flexibility: If you’re doing “real” science,
off-the-shelf software will fail you every time▫80% rule…
![Page 14: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/14.jpg)
Don’t Try This With Excel
•Millions of reads compared against mouse transcriptome• Determining number of distinct species and frequency of members in each• Summarize using plots for each codon
![Page 15: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/15.jpg)
Remember SQL?
![Page 16: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/16.jpg)
The Quest For Power
![Page 17: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/17.jpg)
Heard at lab meeting:
“I would have shown you this graph
but Excel crashed while computing a big file”
→You can’t do this (censored) on your laptop anymore
![Page 18: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/18.jpg)
Welcome To The Cloud
![Page 19: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/19.jpg)
Why Own When You Can Rent?
![Page 20: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/20.jpg)
An Example: PathSeq•Compare millions of short-read sequences
against all genomic + transcriptomic sequences for all microbes (!)
Amazon Cloud “Management Console”
![Page 21: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/21.jpg)
Why The Cloud Matters For Biologists
• You can purchase as much computing power as you need▫You don’t have to run/manage what you don’t use
• Your purchasing computing power, not machines▫ never outdated
• Can easily migrate from one machine type to another (minutes)
• Can add storage in seconds• Accessible from anywhere• Easy to share e.g., (large) datasets with others
![Page 22: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/22.jpg)
![Page 23: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/23.jpg)
04/10/2023
23
WEKA: the software
•Machine learning/data mining software written in Java (distributed under the GNU Public License)
•Used for research, education, and applications•Complements “Data Mining” by Witten & Frank•Main features:
▫Comprehensive set of data pre-processing tools, learning algorithms and evaluation methods
▫Graphical user interfaces (incl. data visualization)▫Environment for comparing learning algorithms
![Page 24: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/24.jpg)
04/10/2023University of Waikato
24
![Page 25: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/25.jpg)
04/10/2023
25
University of Waikato
Explorer: building “classifiers”
•Classifiers in WEKA are models for predicting nominal or numeric quantities
•Implemented learning schemes include:▫Decision trees and lists, instance-based
classifiers, support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, …
•“Meta”-classifiers include:▫Bagging, boosting, stacking, error-
correcting output codes, locally weighted learning, …
![Page 26: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/26.jpg)
04/10/2023University of Waikato
26
![Page 27: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/27.jpg)
04/10/2023University of Waikato
27
![Page 28: There’s No Avoiding It: Programming Skills You’ll Need](https://reader034.fdocuments.in/reader034/viewer/2022042714/54bbc0514a795938428b4591/html5/thumbnails/28.jpg)
04/10/2023University of Waikato
28