Post on 07-Jan-2017
Department of GeographySchool of Social Science & Public Policy
THE FULL STACKJON READES
OBJECTIVE
To provide an overview of the tools and technologies that I have found – or seen – to enable good development practice & productive research.
MY BACKGROUND
BA in Comparative Literature in 1997.Went to work for dot.com start-up.Learned to program, on the job.Learned SQL, on the job.Learned to back up more often, on the job.Managed sites, ETL systems & analytics over many years.Re-entered academia in 2006.PhD at CASA; collaboration with SENSEable City lab.Lecturer at King’s since 2013; helped set up Geocomputation pathway.
Net result: only
Lit grad with an
Erdös number?
Not this.
MOTIVATION
HOW DOES ‘BIG DATA WORK’ WORK?
Idea
Exploration
DevelopmentRevision
Writing Up Start at random
point &
repeat many, many
times.
BIG DATA WORK ON A PRACTICAL LEVEL
MY EXPECTATIONS FOR (GOOD) TOOLS
They must be useful when I need them.They must get out of the way when I don’t.They must fail gracefully when they can’t help it.They must play well with other tools where feasible.They must make it easy for me to do the right thing.They should grow gracefully into operational systems.
Very few tools
do all of these
well.
WHERE DO WE GO FROM HERE?
In the remainder of this talk I will try to link my outputs – the pretty pictures – to the process by which they were created.If you want to know more about something you see, just stop me.
Considerations: Coherence of syntax Coherence of libraries Data-munging features Spatial analytic support Map-making & data viz Ability to get things done Availability of a good IDEBut it’s really the ‘value added’ features that matter.
PROGRAMMING LANGUAGES
Cellular Census (2007)
Considerations: Standards compliance (Spatial) Feature set (esp.
indexing) Replay/Logging Replication & distribution Access controls & user
managementA lot can be done without spatial queries. Learn about indexing, query & schema design, and partitioning.
DATA STORAGE & MANAGEMENT
The ‘Big Bubble’? (2014)
Considerations: Ease-of-use Scriptability Ability to layer InteroperabilityDistinguish between mapping to communicate results with a spatial dimension and mapping to produce actual maps?
GEODATA VISUALISATION
Global Health Partnerships (2016)
Considerations: Collaboration Scalability Ease of recovery Scale of useBest if you never learn SVN/CVS, then your brain will not be done in by Git.
VERSION CONTROL & RECOVERY
Oyster Card Work (2012)
Git: commits on
a plane!
Considerations: Getting out of the way Compatibility Collaboration Editing & comments Quality of outputWhat helps you to think? What helps you write first, but makes formatting later easy?
WRITING
Thesis & ‘Space of Flows’ (2011, 2014)
Considerations: How easy to
backup/share? How often? Where stored? How easy to recover? How selective is
recovery?Backup early & backup often. Never trust one solution or one location. Note: data protection issues.
BACKUP & REPLICATION STRATEGIES
Pint of Science (2014)
Considerations: Performance Encryption ACLs
(users/groups/systems) Password ManagersEncrypt! Encrypt! Encrypt! Encourage use of password managers.
COMPLIANCE & DATA SECURITY
Also worth watching: Travis CI: automated
testing with GitHub integration.
Docker/Vagrant: replication & virtualisation.
Full replication of someone else’s entire data analysis process is harder than you think!
REPLICABLE RESEARCH
N/S Housing Divide (2017?)
WHAT’S MISSING?
• Better ways of specifying the full analytical ‘context’ – including versions of libraries, platform, etc. – as well as the input/output ‘pipeline’ – such as data and results (rctrack seems to want to do this, but only with R, YAML more promising).
• Ways of talking about data processing pipelines & steps (UML is not the answer).
• Valuing of good (open) code & good data by institutions and research councils.
THE BIG PICTURE
Tools (ca. 2006): Eclipse Perl/Java Oracle 8i Cron jobs OLAP Tools CVS ArcMap
Tools (ca. 2016): R/Rstudio Python Postgres + PostGIS Cron jobs Knitr, etc. Git QGIS
THE BIG PICTURE
Massive shift from expensive proprietary to cheap open (both software & hardware).Underlying distinction between operational and development/research environments persists. The problem: one tends to evolve into the other.
FINAL THOUGHT
Document your code. And any sources it drew upon.You will regret not doing it.
THANK YOU
Jon Reades@jreades reades.comkingsgeocomputation.org