Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a...
Transcript of Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a...
![Page 1: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/1.jpg)
Engineering
October 2019
Fast, safe, and reliableThe future of configuration
Qui NguyenTech Lead
![Page 2: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/2.jpg)
How can we best empower developers?Enable them to make changes as quickly as possible, while making it easy to stay safe.
![Page 3: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/3.jpg)
Configuration
Values used internally by applications that you want to change over time.
![Page 4: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/4.jpg)
Decouple the application from resource names like S3 buckets or logs.
Tune the number of retries and timeouts for a request.
Change the number of search results.
Update percentages or whitelists for an experiment.
![Page 5: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/5.jpg)
Constants
config.py
---------
TIMEOUT_SEC = 10
client.py
---------
from config import TIMEOUT_SEC
do_request(TIMEOUT_SEC)
+ Reusable
– Need to update code
– Can't vary across environments
![Page 6: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/6.jpg)
Environment variables
client.py
---------
timeout = os.environ.get(
'TIMEOUT_SEC'
)
do_request(timeout)
$ TIMEOUT_SEC=10 python client.py
+ Reusable
+ Can vary across environments
– Need to restart process
![Page 7: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/7.jpg)
Read from files
client.yaml
-----------
timeout: 10
client.py
---------
load_file_if_changed('client.yaml')
timeout = get_config('timeout')
do_request(timeout)
+ Reusable
+ Can vary across environments
+ Get new value if file changes
![Page 8: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/8.jpg)
srv-configs
git master
$ git push origin i/qui/timeout ...remote: merged to master
![Page 9: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/9.jpg)
srv-configs
● Files from git copied to client hosts every minute, with rsync
git master
replica replica
client client client client
![Page 10: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/10.jpg)
srv-configs
Varying files by environment
● Configs stored in git for all environments
● Clients materialize a view based on their location ("hiera-merging")
common/ foo.yamlecosystem/ prod/ foo.yaml dev/region/ ...
![Page 11: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/11.jpg)
srv-configs
server1ecosystem: prod
-> region: uswest1
foo.yaml
log_name: foo_errorsmax_timeout_secs: 1
common/foo.yaml
log_name: foo_errorsmax_timeout_secs: 5
ecosystem/prod/foo.yaml
max_timeout_secs: 1
![Page 12: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/12.jpg)
srv-configs
git master
$ git push origin i/qui/timeout ...remote: merged to master
I need to decrease that default timeout.
![Page 13: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/13.jpg)
See changes in 2 minutes.Push a branch and your service will start using those values as soon as the files update, with no restart required.
![Page 14: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/14.jpg)
Configuration brings down the site
experiments:
buttons:
active: True
cohorts: [...]
-----
cohort = get_cohort('buttons')
# modern_blue, layout_blue
use_blue_buttons = 'blue' in cohort
![Page 15: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/15.jpg)
Configuration brings down the site
experiments:
buttons:
active: False
cohorts: [...]
-----
cohort = get_cohort('buttons')
# modern_blue, layout_blue
use_blue_buttons = 'blue' in cohort
TypeError: argument of
type 'NoneType' is not
iterable
![Page 16: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/16.jpg)
Incidents srv-configscode changes
![Page 17: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/17.jpg)
srv-configs deploy
git master
$ git push origin i/qui/timeout ...remote: deploy started
![Page 18: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/18.jpg)
Make it easy to do the right thing.Automatically stage changes, from stage to canary to production.
Roll back in one step.
![Page 19: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/19.jpg)
![Page 20: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/20.jpg)
Interactive interface
Progress of deploy updated as it goes through stages.
1
1 Prominent button to encourage rolling back in case of issues.
2
2 More details for authors in thread.
3
3
![Page 21: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/21.jpg)
git master
$ git push origin i/qui/timeout ...remote: deploy started
--- a/foo.yaml+++ b/foo.yaml@@ -1,2 +1,2 @@ log_name: foo_errors-max_timeout_secs: 5+max_timeout_secs: 2
![Page 22: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/22.jpg)
Implementing staged deployments
When an integration branch (i/) is pushed, git hooks:● Generate a deployment version of each affected file● Update control file with rollout information for that branch
![Page 23: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/23.jpg)
Implementing staged deployments
main/foo.yaml
log_name: foo_errorsmax_timeout_secs: 5
deploy/foo.yaml
log_name: foo_errorsmax_timeout_secs: 2
branches.json
"i/qui/timeout": { "stage": 1.0, "canary": 0.0, "production: 0.0,}, ...
Generated by git hooks and synced to clients
![Page 24: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/24.jpg)
New clientlib
Provides an interface between data (files) and logic
Before load_file('my_service.yaml') watcher('my_service.yaml').reload_if_changed()
After load_namespace('my_service')
get_watcher('my_service').reload_if_changed()
Consider deploy files and branches.json!
![Page 25: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/25.jpg)
Rolling back
When the rollback button is clicked:● Delete deploy files● Remove branch from branches.json
![Page 26: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/26.jpg)
![Page 27: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/27.jpg)
Rolling back later
Tell the user how to rollback after the deploy is done, too.
![Page 28: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/28.jpg)
Automatic rollbacks
Can monitor SLOs or other metrics in deploy process● Notify users and default to
rolling back
![Page 29: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/29.jpg)
This is cool...
but why files?
![Page 30: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/30.jpg)
We should switch to a datastore!
Clients can watch for changes, instead of rsyncing everything
More natural API for automated changes● Updating deploy progress● Updating config from other systems
![Page 31: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/31.jpg)
…or not. Files have a lot of features.
Can take advantage of existing workflows for code files● Editing with text editors● Searching, reviewing
Version control (git)● Viewing history● Reverting changesets
Easy to scale and handle master failures
![Page 32: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/32.jpg)
Time investment
Impact on developers
DatabaseFiles
![Page 33: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/33.jpg)
![Page 34: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/34.jpg)
Configs are as powerful as code.
![Page 35: Fast, safe, and - USENIX...When an integration branch (i/) is pushed, git hooks: Generate a deployment version of each affected file Update control file with rollout information](https://reader036.fdocuments.in/reader036/viewer/2022070815/5f0eca4f7e708231d440f364/html5/thumbnails/35.jpg)
How can we best empower developers?Dynamically loaded configuration from files allows developers to spend less time updating and deploying code.
Interactive deploy interface provides rich feedback and guides developers on best practices to keep the site safe.