Calculated risk - a scaleable management technique or just risky business?
Zero to scaleable in ten minutes
-
Upload
matt-walters -
Category
Technology
-
view
138 -
download
2
Transcript of Zero to scaleable in ten minutes
![Page 1: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/1.jpg)
Zero to scaleable in ten minutes
Less than obvious tools & tips for writing software that lasts
Matt Walters @mateodelnorte [email protected]
![Page 2: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/2.jpg)
What do you mean… ‘scaleable’?
As businesses grow, their understanding of problems they’re solving also grows
Scaleable = easy to understand, maintain, and add to
During that growth, software changes. Good software is easiest to change, while still getting
things done.
![Page 3: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/3.jpg)
Combining standard, available tools, and repeating simple, predictable patterns during development will keep your code understandable, maintainable, and easily improved.
Our toolkit will be: • gitslave • tmuxinator • rabbitmq • servicebus *and some very simple patterns
How do we scale an architecture?
![Page 4: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/4.jpg)
gitslaveWhy have a monolith, when you can have a
metarepo?
> man gits
Gitslave Home Page: <http://gitslave.sf.net>
NAME gits - The git slave repository tool for multi-repository management
SYNOPSIS gits [-p|--parallel COUNT] [-v|--verbose]+ [--quiet] [--help] [--version] [-n|--no-pager] [--paginate] [--eval-args] [--exclude SLAVE-REGEXP] [--keep-going] [--no-commit] [--no-hide] [--no-progress] [--no-master] [--with-ifpresent|--just-ifpresent] SUBCOMMAND [ARGS]...
OVERVIEW gits is a program that assists in assembling a meta-project from a number of individual git repositories which operate (when using gits) as if they were one git repository instead of many, similar to the way that CVS works by default and svn (v1.5) can be coerced to work. Some of these individual git repositories may be part of other meta- projects as well.
![Page 5: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/5.jpg)
gitslaveWhy have a monolith, when you can have a
metarepo?
gitslave allows you to have all of your code in one location, like a monolithic app, while allowing fine grained updates of each of your solution’s component projects.
one ‘gits pull origin master’ gets all new changes, for all related projects / repositories, while you can still perform fine grained operations on any particular repo.
![Page 6: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/6.jpg)
gitslaveWhy have a monolith, when you can have a
metarepo?
> ls -ladrwxr-xr-x 18 mateodelnorte staff 612 Jul 14 00:52 . drwxr-xr-x 77 mateodelnorte staff 2618 Jul 19 23:01 .. -rw-r--r-- 1 mateodelnorte staff 411 Jul 12 22:22 .eslintrc.js drwxr-xr-x 12 mateodelnorte staff 408 Jul 20 11:34 .git -rw-r--r-- 1 mateodelnorte staff 59 Jul 14 00:25 .gitignore -rw-r--r-- 1 mateodelnorte staff 127 Jul 14 00:26 .gitslave -rw-r--r-- 1 mateodelnorte staff 2849 Jul 5 15:56 Makefile drwxr-xr-x 16 mateodelnorte staff 544 Jul 14 01:18 denormalizer drwxr-xr-x 16 mateodelnorte staff 544 Jul 10 17:51 my-service-one drwxr-xr-x 18 mateodelnorte staff 612 Jul 14 00:39 my-service-two drwxr-xr-x 18 mateodelnorte staff 612 Jul 14 00:39 my-site -rw-r--r-- 1 mateodelnorte staff 1184 Jul 19 23:32 tmuxinator.all.yml
![Page 7: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/7.jpg)
gitslaveWhy have a monolith, when you can have a
metarepo?
> cat .gitignore/denormalizer/ /my-service-one/ /my-service-two/ /my-site/
![Page 8: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/8.jpg)
gitslaveWhy have a monolith, when you can have a
metarepo?
> cat .gitslave
"../denormalizer" “denormalizer" “../my-service-one“ “my-service-one“ "../my-service-two" “my-service-two“ “../my-site" “my-site"
![Page 9: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/9.jpg)
gitslave
brew install gitslave
![Page 10: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/10.jpg)
tmuxinatorHeads up. You need a good view of your system.
![Page 11: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/11.jpg)
tmuxinator.yml-based config-driven tmux. that’s it.
name: my-system root: ./
windows: - site: layout: tiled panes: - my-site: - printf ‘\033]2;my-site\033\\' - cd ./my-site - make run - services: layout: tiled panes: - my-service-one: - printf '\033]2;my-service-one\033\\' - cd ./my-service-one - make run - my-service-two: - printf '\033]2;my-service-two\033\\' - cd ./my-service-two
=
![Page 12: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/12.jpg)
tmuxinatorHeads up. You need a good view of your system.
Panes per process
![Page 13: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/13.jpg)
tmuxinatorHeads up. You need a good view of your system.
Windows per context, business domain, etc
![Page 14: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/14.jpg)
tmuxinator
gem install tmuxinator
![Page 15: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/15.jpg)
Neat whats next?We’ve talked about where you’ll house your project, and
the different sub-projects within it. (hint: one gitslave metarepo and n project repos)
Let’s talk a bit about messaging.
And we’ve talked about how you’ll stare at it while it’s on your screen.
(hint: tmuxinator)
![Page 16: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/16.jpg)
Why messaging?
Your processes need an interface
![Page 17: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/17.jpg)
Why messaging?
Your processes need an interface
REST
![Page 18: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/18.jpg)
Why messaging?
Your processes need an interface
REST ?What happens when a process dies?
![Page 19: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/19.jpg)
Why messaging?
Your processes need an interface
REST ??What happens when a process dies?
![Page 20: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/20.jpg)
Why messaging?
Your processes need an interface
![Page 21: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/21.jpg)
Why messaging?
Your processes need an interface
Shared DB
![Page 22: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/22.jpg)
Why messaging?
Your processes need an interface
Shared Spaghetti
Doable, but usually unclear which process ‘owns’ database structure and data. Can lead to maintenance hell.
![Page 23: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/23.jpg)
Why messaging?
Your processes need an interface
![Page 24: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/24.jpg)
Why messaging?
Your processes need an interface
Queues
![Page 25: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/25.jpg)
Why messaging?
Your processes need an interface
Queues
?Processes go down with queues…
![Page 26: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/26.jpg)
Why messaging?
Your processes need an interface
Queues
And come right back up with no message loss.
![Page 27: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/27.jpg)
Why messaging?
Your system gets a safety net
Queues
?x5Added benefit: Planned human intervention for errors.
![Page 28: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/28.jpg)
rabbitmqmessaging that just works
• direct send to queue • fanout / topic routing • highly available • highly performant • used in financial exchanges, industrial applications and more • open source • free
![Page 29: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/29.jpg)
rabbitmqmessaging that just works
brew install rabbitmq
![Page 30: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/30.jpg)
servicebussuper simple messaging in node
• direct send • pub / sub / fanout / topic -routing • simple to set up • highly performant • used in financial exchanges, online advertising and more • open source • free • perfect for creating microservices!
![Page 31: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/31.jpg)
servicebussuper simple messaging in node
npm install servicebus —save
![Page 32: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/32.jpg)
servicebussuper simple send+listen messaging in node
// process 1 var bus = require('servicebus').bus(); bus.listen('my.event', function (event) { console.log(event); });
// process 2 var bus = require('servicebus').bus();
setInterval(function () { bus.send('my.event', { my: 'event' }); }, 1000);
![Page 33: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/33.jpg)
servicebussuper simple pub+sub messaging in node
// process 1 var bus = require('servicebus').bus(); bus.subscribe('my.event', function (event) { console.log(event); });
// process 2 var bus = require('servicebus').bus();
setInterval(function () { bus.publish('my.event', { my: 'event' }); }, 1000);
// process 3
![Page 34: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/34.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js const bus = require(‘servicebus').bus(); const create = require(‘./lib/create’); bus.listen(‘order.create', function (event) { create(event); });
![Page 35: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/35.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js const bus = require(‘servicebus’).bus(); const retry = require(‘servicebus-retry’); bus.use(retry({ store: new retry.RedisStore({ host: process.env.REDIS_HOST, port: process.env.REDIS_PORT }) })); const create = require(‘./lib/create’); bus.listen(‘order.create', function (event) { create(event, (err) => { if (err) return event.handle.reject(); event.handle.ack(); }); });
![Page 36: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/36.jpg)
(micro)servicebussome simple patterns get us going
// order-svc refactored bus to ./lib/bus.js const bus = require(‘servicebus’).bus(); const retry = require(‘servicebus-retry’); bus.use(retry({ store: new retry.RedisStore({ host: process.env.REDIS_HOST, port: process.env.REDIS_PORT }) })); module.exports = bus;
![Page 37: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/37.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’); // moved bus initialization to own module const create = require(‘./lib/create’); bus.listen(‘order.create', function (event) { create(event, (err) => { if (err) return event.handle.reject(); event.handle.ack(); }); });
![Page 38: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/38.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’); // moved bus initialization to own module const create = require(‘./lib/create’); bus.listen(‘order.create', function (event) { create(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); });
service publishes to the world when it’s done!
![Page 39: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/39.jpg)
commands tell services when an actor wants an action
client send commands to instruct a service to do work
commands are sent async, fire and forget
commands are sent directives: order.create
web app order-svc
order.create!
![Page 40: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/40.jpg)
events tell the world when you’re done
services publish when done performing work
services publish messages to any services that wish to listen
events are sent past-tense: order.created
order-svc fulfillment-svc
order.created!order.created!
![Page 41: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/41.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’); // moved bus initialization to own module const create = require(‘./lib/create’); bus.listen(‘order.create', function (event) { create(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); });
![Page 42: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/42.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’); // moved bus initialization to own module const cancel = require(‘./lib/cancel’); const create = require(‘./lib/create’); bus.listen(‘order.create', function (event) { create(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.created’, order); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); }); bus.listen(‘order.cancel', function (event) { cancel(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.canceled’, order); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); });
code buildupincreases
cognitive load
![Page 43: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/43.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’); // moved bus initialization to own module require(‘./lib/handlers/orderCanceled’); require(‘./lib/handlers/orderCreated’); // moved handlers to their own modules, // hiding their complexity from other code
![Page 44: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/44.jpg)
(micro)servicebussome simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’); const log = require(‘llog’); const registerHandler = require(‘servicebus-register-handlers’); registerHandlers({ bus, handleError: function handleError (msg, err) { log.error(`error handling ${msg.type}: ${err}. rejecting message` + `w/cid ${msg.cid} and correlationId ${this.correlationId}.`); log.fatal(err); msg.handle.reject(() => { bus.close(); process.exit(1); }); }, path: ‘./lib/handlers', // handlers are now one listen or subscribe per file queuePrefix: 'order-svc', });
![Page 45: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/45.jpg)
Show the stuff!
quick show and tell
![Page 46: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/46.jpg)
Let’s recap!‘cause there was a lot!
1. gitslave lets you structure a project like a monolith, but have the convenience of many repositories
2. tmuxinator gives you a head’s up view of your system, all running locally
3. messaging provides a durable, convenient interface between your processes
4. rabbitmq + servicebus enable easy send+listen & pub+sub messaging patterns between processes
5. processes send commands and publish events 6. simple, expected, structure in your services keeps
cognitive load low, and projects easy to maintain
![Page 47: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/47.jpg)
Thanks!We’ll cover more on event-driven architecture, microsvc, CQRS, and
more in a future talk!
![Page 48: Zero to scaleable in ten minutes](https://reader030.fdocuments.in/reader030/viewer/2022020314/587582c71a28ab78498b7b37/html5/thumbnails/48.jpg)
Zero to scaleable in ten minutes
Matt Walters github & twitter: @mateodelnorte email: [email protected]
https://github.com/tmuxinator/tmuxinatorrabbitmq.com
http://gitslave.sourceforge.net/
https://www.npmjs.com/package/servicebus