How to utilise
37
Transcript of How to utilise
Alexander Tolstikov Smile Open Source Solutions
How to utilise Open Source tools to create CI/CD & DevOps workflows
without vendor lock-in
Possible workflows:
- Create Kubernetes cluster- Build release artifact (composer install, yarn build, npm)
- Deliver artifact (ansistano, capistrano, etc)- Drush operations
- Automated functional / load / smoke testing - Anything else?
Workflow example:Kubernetes cluster creation on GCP
Evolution of the workflow
Iteration 1:Create k8s cluster manually with UI
ToolsGCP UI
Secrets/Access You need to be authenticated in the browser
Pros • Very fast • No need to maintain dependencies / tools on the local PC
Cons • Need to remember all command params • Need to do the same for all resources (VPC, subnets, DNS, etc) • Human error-prone • Only you know what you did
Congratulations! We just created a snowflake cluster!
Snowflakes are beautiful…
…but so different (and fragile)
“ “Martin Fowler
The true fragility of snowflakes, however, comes when you need to change them. Snowflakes soon become hard to understand and modify. You're not sure what parts of the configuration are important. Their fragility leads to long, stressful bouts of debugging.
Toolsgcloud (from Google Cloud SDK)
Secrets/Access Need to be authenticated on the local PC
Pros More “fixed” compared to manual UI operation
Cons • Need to remember all command params • Need to do the same for all other resources (VPC,
subnets, …) • Human error-prone • No centralised history / logs • Still “Snowflake” infrastructure
Iteration 2: Create k8s cluster manually with “gcloud” command
Tools • terraform Secrets • Need to use GCP key file stored on the local PC Principles / Concepts • Infrastructure as Code Pros • Logic and configuration are separated (different terraform files) • Can be (should be!) checked-in into Git repo Cons • Need to copy/paste files with variables for different environments manually • Configuration drift is still possible with multiple environments
Iteration 3:Create k8s cluster with terraform: Manual terraform commands execution
Iteration 4:PROD & TEST environments & single master config file
Need to introduce some tools!
“ “Unix philosophy
• Do one thing and do it well. • Make programs to work together. • Write programs to handle text streams,
because that is a universal interface.
Tool: uniconf (https://github.com/aroq/uniconf)Config processing (YAML)
Sorry, no documentation…
Input config Output config
OutputInput
Tool: uniconf (https://github.com/aroq/uniconf)Config processing (YAML)
Tool:jq (https://github.com/stedolan/jq)
Config processing (JSON)
Use to extract data from JSON
A lot of filters are available
Check the documentation!
Tool:yq (https://github.com/mikefarah/yq)
Config processing (YAML)
Same as jq tools but for YAML files
Basically is a wrapper over jq tool
uniconf | yq
Output
Tool:gomplate (https://github.com/hairyhenderson/gomplate)
Template rendering. Supports lots of local and remote datasources.
Check the documentation!
Output
Output
Template
Tool:variant (https://github.com/mumoshu/variant)
“One glue to glue them all!”
“Makefile on steroids”
Task definition
Task execution
Variant task execution
Plain command vs Variant task execution
Iteration 4:PROD & TEST environments & single master config file
Now you need to maintain more tools / dependencies:
terraform unicorn
jq / yq gomplate
variant
It can be an issue if we want to execute this workflow somewhere else because we’ll need to sync all software versions, dependencies, etc
Dependency hellPythonRuby
NodeJS…
How to fix the Dependency Hell issue: Put all tools into the container (Docker)
+ =love
Secrets management
Managing secrets
Git Secret (https://github.com/sobolevn/git-secret#git-secret)
Chamber (https://github.com/segmentio/chamber)
GPG (https://gpgtools.org/)
AWS Vault (https://github.com/99designs/aws-vault)
Execute the workflow in the CICD engine: Gitlab CI
Gitlab CI pipeline
Demo time!
Next steps:GitOps? Check Atlantis:
https://github.com/runatlantis/atlantis
Alexander [email protected] / [email protected]
You can find me on Drupal slack as well!
https://www.smile.eu/
