Deploying an Open Source DAM in SAAS Mode (European Drupal Days 2015)
-
Upload
ibuildings-italia -
Category
Internet
-
view
110 -
download
0
Transcript of Deploying an Open Source DAM in SAAS Mode (European Drupal Days 2015)
© Ibuildings 2014/2015 - All rights reserved
#DrupalDaysEU
Deploying an Open Source DAM in SAAS Mode Michel van de Ven
MediaMosa
● Drupal-based Digital Asset Management system
● Commissioned by SURFnet and Kennisnet in 2007
● Open Source (GPLv2), Open Standards
● Webservice oriented: REST
● Store & retrieve assets
● Manage metadata using open standards
– Dublin Core, Qualified DC, IEEE/LOM, W3C Ontology
– OAI-PMH, CQL (SRU/SRW), BagIt
● Transcode video / audio (FFmpeg), images, PDF &c, stream content
● Storage on disk, SAN/NAS, S3
● Drupal modules for connection mgmt, media and views (by Acquia)
● Users: Kennisnet, NIBG, UGent, Uni Leuven, UvA, TiU, RUG, Digipolis Antwerp, UOslo, Avans, Politie Academie, Acquia (NBC Sports), Berkeley, Cineca &c
● http://mediamosa.org & MediaMosa Foundation
MediaSalsa infrastructure (simplified)
● For each DTAP environment
– Backends: Core MediaMosa services
– Frontends: Optional, Drupal, Java, Python, apps, ...
– Web servers
– Database servers
– Solr servers
– Transcoding servers
● Instance @ RUG DC for NL .edu
● Instances @ Commercial DCs for non-edu
Inuits
● Inuits is an Open Source company
– We contribute back
● 50+ people in 4 locations (BE, NL, UA, CZ)
● One language: English
● We offer
– Consulting
– Development
– System Administration– and a nice SaaS Platform
Distributed team
● Daily virtual stand-up over XMPP / Hangout
● Redmine project management
– Issues, tasks, sprints
– Repositories
– Documentation (wiki)
– Used by customers as well
● Internal mailing lists
● Notifications by mail and XMPP
● Internal training, all the time
Dev and Ops
● Dev
● Architecture
● Develop new features
● Write unit tests (!)
● Ops
● Infrastructure (Puppet)
● CD and CI (Jenkins)
● Monitoring
→ Ops teach dev about monitoring and distributed services, scale, redundancy, failover, high availability, systems engineering, ...→ Dev teach ops about required libraries and testing, dependencies, daemon tuning, software engineering & architecture, ...
Continuous D.*
● Unit Tests + Continuous Integration →
● Continuous Delivery vs Continuous Deployment
– “Continuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time” (@ccaum)
● Puppet code
– Deployed to dev environment
– Same puppet code for each environment
– User-triggered deployments to UAT & production
– Feature flags in Puppet code per environment (switchable architecture)
● Application code
– Continuous integration in dev
– User-triggered deployments to UAT
– Deployment to production is a business decision
Testing
● Developers test a lot, but
– The tests don’t work
– It works on my machine™
– Wrong platform
– Wrong PHP version
Fixed now, thanks to Jenkins!
and Vagrant / $container_tech of the week
Version Control
● Git
● Code is under revision control
– Prefer small commits
– Local feature branches
● Infrastructure as code → git / hiera
next: Software Defined Infrastructure
Use the dist native packaging system
● Consistency, security, dependencies
● Uniquely identify where files are coming from
● Source repo may not be reacheable
● Little overhead when you automate
● Configuration does not belong in a package
Pipelines
● A collection of jobs
● Run in sequence
● Start on checkout, end at deployment
● From the developers’ side:
→ Git push
← Mail with changes + link to deploy / promote
Our current pipelines
● Puppetized
● Puppet code and Application code (backend and frontends)
● 1 pipeline per project phase / sprint
Pipelines steps
● Checkout● Syntax: php -l● Style: Drupal Coder● Package: FPM● Deploy to dev environment: mcollective● Tests in dev environment: drush run-tests● Publish package and promote: mcollective
Promotion
● At the end of the pipeline● Send a simple email
● A link to the promotion page● The changelog
● Promotion page contains one button per environment● Must promote to UAT before Production
Tools used with Jenkins
● Pulp to manage RPM repositories
– Collect, process, serve / push out
● Mcollective to update packages, run drush
– Control and update marionettes
Logstash
● Collect all the logs
– Drupal logs
– Apache logs
– Deployment logs
– System logs
● Interpret, filter and correlate them
● Logstash, ElasticSearch, Kibana, statsd, Graphite
Graphite + gdash
● Collectd
● Monitor platform usage
● FFmpeg usage
● Number of accounts
● Pipelined !
Open Source
● Mediamosa is Fully Open Source
● Lots of the PuppetCode to deploy it is Open Source
● Our passwords etc aren't
Experiences
We have just shared our experience...
But also at the NLUUG Fall Conference, DrupalDays Kyiv, DrupalCon Prague, FlossUK, ElastiCon SF, ...
ContactContactKris Buytaert <Kris Buytaert <[email protected]@inuits.be>>
Inuits Chief Trolling O�cerInuits Chief Trolling O�cer
Further ReadingFurther Reading@krisbuytaert @krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/
Inuits HQ Inuits HQ Antwerp, BelgiumAntwerp, Belgium+32 475 961221+32 475 961221
Branches:Branches:Rotterdam NLRotterdam NLKyiv UAKyiv UAPrague CZPrague CZ