Freedom In Disguise!

Khushil Dep @khushil

www.daemondreams.co.uk

This presentation is NOT …A detailed description of how the MailOnline uses cfengine

Comparing cfengine to puppet/chef/salt/ansible etc

Tutoring you on configuration management - that’s easy and it’s well understood

Selling you any tool or product mentioned herein

Providing you the answers to all your questions

Telling you what tools you should/shouldn’t be using

Regurgitating the last five years of rigmarole

Ruby

This presentation is about …

FINDING a definition for an autonomic cloud platform

EXAMINING our definition of systems engineering

EVOLVING our understanding of operations engineering

LEARNING cfengine and how to use and abuse it

EXPLORING the promise of autonomic machine operation

What does an autonomic cloud platform look like?

Maximises the performance available from the underlying hardware to increase ROI.

Scales robustly and easily both vertically and horizontally, programmatically

Provides durable, secure, fast and reliable storage.

Allows deep, full stack introspection easily and without restriction.

Protects your instances from the effects or attacks of others on the same platform.

Provides a flexible service management toolset.

Defines an open set of API’s which allow access to the full range of capabilities and data.

Recommended Technology StackSmartOS operating system based on Illumos kernel with ZFS, DTrace, SMF and Zones.

Based off the Illumos OpenSolaris fork.

Joyent Public Cloud for a real alternative to Amazon Web Services.

More horsepower for your dollar/pound/euro than AWS in my experience. YMMV.

Engineering excellence - Bryan Cantrill, Brendan Gregg, Ben Rockwood to name a couple of greats who work there.

The Halcyon DreamUser defined software managed by release manager.

cf-engine allows the machine to manage itself under our constraint models.

DTrace provides diagnostic and performance feedback across the stack.

JoyentSDC APIs provide easy functionality for all platform operations.

Systems Engineering is the application of technical expertise, diligence,

reflection, communication, collaboration, patience and innovation

between multi-disciplinary teams to create something of use.

What’s wrong with DevOps?

DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organisation rapidly produce software products and services.

What we do to our machines at the moment …

Imperative approach to machine operation

Obligatory behavioural model where we force or

coerce our machines into actions

Unsympathetic to real-time environmental events

Scales inefficiently introducing risk

You begin to doubt your machines

You being to doubt yourself

What we think we’d like to do with our machines…

What our platforms end up feeling like …

What we should be getting our machines to do …“I’m sorry Dave, I can’t do that…. right now. It would be unwise. I

am aware of events in realtime that would affect the outcome of

your request adversely that you are not aware of.

Don’t worry, I promise to do it as soon as the probability of

success has improved and will orchestrate the dependant

activities so you reach your goals.

There is a lot else you could be doing right now - you don’t have

to wait for me Dave.

I’ve got you Dave. We are a team.

Did you see the game last night?”

What is Operations Engineering anyway …

Enable better machine/human relationships through evolving theory of mind, communication and trust

Theory of Mind (ToM)

The ability to attribute mental states to oneself and others and to

understand that others also have mental states that may differ

from one’s own. The basis of empathy some might say.

Communication

The activity of conveying information through an exchange of

thoughts, messages or information as by speech, visual signals,

writing or by behaviour. It is the meaningful exchange of

information between two or more entities.

Meaningful Information

Knowledge communicated or received concerning a

particular fact or circumstance, expressed with clarity,

within context, concisely, in a timely manner,

significant and with purpose.

TrustSocially we require trust when operating on, and often, beyond the edge of what is known

through practical experience and that which may arise from new possibilities.

!

Psychologically human trust is believing that an entity that is trusted will do what is expected of

it by you.

!

We must trust our machines to operate not only within known boundaries but those that we

cannot yet envisage.

The Machine

Electro-mechanical computation engine with mechanisms for input and output.

Capable of sustained activity without distraction nor deviation.

Fast, accurate, reliable and repeatable task handling.

No capacity for independent imagination.

Able to generate and analyse vast quantities of information.

Inefficient and communicating with humans.

I, HumanBiological system with facilities for input and output.

Incapable of sustained activity without distraction and deviation.

Slow, prone to error, unreliable and fragile task handling.

Excessive capacity for independent imagination.

Unable to generate and analyse vast quantities of information.

Inefficient and communicating with anyone or anything.

Why we fail our Machines

Human process is based on human understanding of events.

Human process is designed for human implementation.

Human process maps dangerously to machine computation and understanding of events.

Humans have hidden sanity checking which machines are unable to deduce or reproduce.

Human imperative command structures do not suit machines which are convergent by nature.

Humans are unable to maintain focus.

VIEW YOUR GOALS FROM THE MACHINES PERSPECTIVE !

YOUR MACHINES CARRY YOUR LOGIC INTO ACTION

Promises

Promises are the foundations of trust between two entities - the machine and the human.

The promiser requires certain promises from the promisee.

Our need for trust in our machines is fulfilled by the machine promising to actuate the

promises we request from it.

We ask that the machine promises these actuations in a timely manner in the machines

context - not our own.

We declare goals and ask the machine to converge on that goal when it’s able to do so.

Promise TheoryProposed by Mark Burgess in 2004.

Autonomy - we do not make assumptions about others behaviour. We only document that behaviour on which we can speak authoritatively. This forces us to more completely define what behaviour we are able to promise and this in turn leads us to a more complete understanding of what are desired state entails.

Emergent Behaviour - when we behave in a model of voluntary cooperation as independent and autonomous agents, certain behavioural patterns must naturally emerge. The atomicity of promises enables us to better understand what we are promising and thus find those contradictions which might else have been missed.

The Forgotten OrchestrationAn operating system is a complex collection of software that orchestrates computer hardware

resources and provides common services for user defined software.

All user defined software sits within this orchestration framework and is already being orchestrated.

It is dangerous therefore to try to further orchestrate user defined software in an obligatory

manner.

Obligatory behaviour is where you force or coerce behaviour from your software or the operating

system with little or no regard to the underlying orchestration.

It’s a bit like the Universe really…

Universal OrchestrationQUARK - packets of energy with mass-like properties which exist in pairs or triplets (we think).

PROTON/NEUTRON - three or more quarks interacting and oh you know, orchestrating!

ATOM - one or more protons, zero or more neutrons and some electrons whizzing about orchestrating the hell out of a snazzy

number!

MOLECULE - made of atoms all orchestrating to their own private adagio!

ORGANELLE - molecules that orchestrate their way to things like a cell nucleus or ribosomes!

CELL - organelles just orchestrating away the Sunday afternoon with things like cytoplasm to make the little things, like the building

blocks of life.

TISSUE - a set of cells orchestrating their way out of boredom to make things like muscle tissue or heart tissue.

ORGAN - a set of at least two types of tissues orchestrating their way through a chorus to something like a heart or a pair of lungs.

Machine Orchestration

Human OrchestrationMicro-management incurs great operational cost and entails an increased risk from unknowable events.

You neither force nor coerce your engineers into behavioural patterns which are alien to them.

You trust your engineers to adapt and use their own preferred behaviour in order to reach set goals.

Your goals might be decided for you by other people or organisations.

These people and organisations trust you to meet the goals and keep your promises.

Why then do you not apply the same trust to your machines?

CFENGINE https://cfengine.com/docs/3.5/manuals.html

What is it?

A systems engineering framework that enables autonomous behaviour of agents.

Created by Mark Burgess in the early 1990’s - the original DevOps tool.

What the hell is DevOps anyway - it’s called Engineering, stop making up words!

Written in C and runs on most unices and even Windows.

Small footprint, very fast execution.

Best of all …

NO RUBY

Autonomic Operation

Machines are best placed to make decisions based on environmental conditions.

Machines do not require obligatory behaviour imposed upon them by humans.

Machines require logic and freedom to enable them to achieve the goals we set them.

This area needs more research and experimentation.

Components of CFenginecf-execd - scheduling daemon which runs cf-agent, gathers output and

send reports.

cf-agent - evaluates policies and actuates changes to the machine.

cf-monitord - samples probes defined in policies and attempts to learn

normal system state.

cf-server - daemon which allows authorised access to policy files and

allow authorised access to cf-runagent.

cf-runagent - connects to a list of cf-server instances and is able to ask

for policy evaluation on these instances foregoing the normal cf-execd

scheduling on these instances

https://cfengine.com/docs/3.5/manuals-components.html

Components of CFengine

cf-hub - collects data about hosts managed by cfengine.

cf-promises - policy validation tool to aid development. Parses policies for

syntax errors. Validates policies composed on multiple files. Validates semantic

correctness of policies. Partially evaluates policies to expose any errors.

Makes NO CHANGES to the system.

cf-key - generates key pairs for remote authentication.

COMMUNITY TOOLSThe following from http://www.cfengineers.net/downloads/cfengine-tools-and-utilities/

§ cf-keycrypt - encrypt/decrypt arbitrary files using cfengine crypto keys for extra security.

§ cf-profile - parses verbose cf-agent execution and records timings and execution trees.

§ cf-runwrapper - cf-runagent wrapper for extra control over cf-runagent behaviour

!

The following from https://github.com/lpefferkorn/cfe-profiler

§ cfe-profiler - measures policy execution times to find top consumers inline with run.

!

The following from https://github.com/cfengine/design-center/tree/master/tools/hcgrep

§ hcgrep - make Hard Classes easier to view and search for

Promise RecapMake a promise about something and cfengine will attempt to keep it.

Each promise is actuated three times to allow convergence to occur.

Everything is a promise and some have commitments:

§ A file exists.

§ It commits to being owned by root

§ A user is present on the system.

§ It commits to having a home directory at /home/khushil

§ The CPU load is below a certain value.

The policies in cfengine are comprised of promises

Convergence is about making the promises to get to the state we want to be in.

What do you need? What must you avoid? Define these promises.

Don’t get distracted by how you get there!

Language conceptsOne grammatical form for all statements in the cfengine DSL.

It is a DOMAIN SPECIFIC LANGUAGE.

Everything in cfengine is made thus.

Promises

Bundles

Bodies

Classes / Classifiers

Variables

Datatypes

bundle bundle_type name { promise_type: classes:: “promiser" -­‐> { "promisee1", "promisee2", ... } attribute_1 => value_1, attribute_2 => value_2, … attribute_n => value_n; }

https://cfengine.com/docs/3.5/manuals-language-concepts.html

How MailOnline thinks about promises

Discovery - examine the system and raise policy defined global classes to

augment hard classes discovered by cfengine.

Contract - select from a set of pre-defined bundles of promises which are

relevant to our desired state for the machine within it’s function context.

Actuation - selected contracts will ensure that our desired state is converged

upon as quickly and safely as possible.

An example desired state to converge uponI want to deploy our snazzy new web application.

nginx is required on the machine.

An application specific nginx configuration is required for nginx.

Our application code must be on the machine

It must be accessible my the nginx service

It must be secure

nginx must be running to serve traffic

If any of these promises are not met we will not reach our goal.

Simple language rulesKeywords, variable names, bundles, bodies and classes must be composed of (a-zA-Z-9_)

Literal data must be quoted

Promise bundles are declared thus:

bundle agent-type identifier { … }

Promise bodies are declared thus:

body constraint_type template_identifier { … }

Body attributes are declared thus:

LHS (cfengine word) => RHS (user defined data)

https://cfengine.com/docs/3.5/reference-syntax.html

Classes / Classifiers HARD classes are discovered by cfengine upon cf-agent execution before any other policies are

converged.

27_0_0_1 64_bit 8_cpus Afternoon

SOFT classes are user defined and used to implement classification and logic.

Evaluated when bundles are evaluated.

Classes are LOCAL to the bundle they are defined in and are NOT accessible outside the bundle.

Classes defined in common bundles ARE accessible as they are GLOBAL classes within the

namespace.

Classes can be raised by promises upon promise outcomes to further aid classification and logic.

https://cfengine.com/docs/3.5/manuals-language-concepts-classes.html

VariablesScalar variables

List variables

Lists can be

slist - strings

ilist - integers

rlist - reals

inf is a special constant representing an unlimited value

“CFEngine typing is mostly dynamic, and CFEngine will try to coerce string values into int and real types, and if it cannot it will report an error.

However, arguments to built-in functions check the defined argument type for consistency.”

https://cfengine.com/docs/3.5/manuals-language-concepts-variables.html

Scalar variables hold single values:

!

vars:  

  “little”   string   =>   “little”;  

  “boy”   int     =>   “4”;  

  “blue”   rela     =>   “3.147”;  

!

$(little) or $(bundle_name.little)

${little} or ${bundle_name.little}

List variables hold several values:

!

Vars:  

  “strings”  slist   =>   {“this”,”is”,”a”,”list”};  

  “ints”     ilist     =>   {“1”,”2”,”3”};  

  “reals”     rlist     =>   {“1.1”,”2.2”,”3.3”};  

!

@(strings) refers to the whole list

$(strings) will loop through each element

Bundles

A collection of promises normally grouped by function or target.

Bundle agent types:

agent

server

monitor

common

Classes and Variables defined in common bundles are GLOBAL in scope

https://cfengine.com/docs/3.5/manuals-language-concepts-bundles.html

PromisesEverything is a promise

Promises have types

files, commands, methods, reports, packages, processes, storage, services, databases, guest_environments, outputs

Promises have bodies

Promises are grouped into bundles

Bundles have types

agent, common, edit_line, server, knowledge, monitor

Bundles live in namespaces

Namespaces have access to global classes

Promises can call modules which are external scripts

Modules can be sent classes or variables on calling

Modules can send classes or variables on exit

https://cfengine.com/docs/3.5/manuals-language-concepts-promises.html

Everything is a promisePromises can be made about different subjects such as command execution, service control, ACL’s.

Promises have types depending on the bundle you’re working in.

Promisers promise and can be any object such as a file or network or even a port.

Promises have attributes which affect the behaviour of the promise.

Implicit promises such as reports or commands have implicit behaviour.

https://cfengine.com/docs/3.5/reference-promise-types.html

Normal ordering

Normal ordering is the sensible order in which behaviours should actuate.

Normal ordering helps maintain equilibrium within a system.

Bundles are actuated strictly in the order dictated by bundlesequence.

The bundlesequence maybe re-ordered by the use of classes.

Agent promises are actuated three times to allow convergence.

https://cfengine.com/docs/3.5/manuals-language-concepts-normal-ordering.html

Looping in CFengine

No explicit loops anywhere to be found!

cfengine uses lists

Referencing a list as a scalar will cause implicit

looping through the list

Powerful behaviour

Demands a shift in thinking

bundle agent implicit_looping { vars: “my_list” slist => {“a”,”b”,”c”,”d”}; !

reports: “$(my_list)”; }

https://cfengine.com/docs/3.5/manuals-language-concepts-loops.html

Multiple list looping in CFengine2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  a  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  b  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  c  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  d  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_rootprocs  is  230.00  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_rootprocs  is  209.30  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_rootprocs  is  150.77  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_otherprocs  is  17.00  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_otherprocs  is  15.47  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_otherprocs  is  11.16  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_diskfree  is  93.00  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_diskfree  is  84.63  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_diskfree  is  61.07  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_loadavg  is  1.36  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_loadavg  is  1.44  

2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_loadavg  is  3.10

bundle  common  control  {     bundlesequence  =>  {"simple_implicit_looping",  "stats_iteration"};  }  !bundle  agent  simple_implicit_looping  {     vars:       "simple_list"   slist   =>   {   "a","b","c","d"   };           reports:       "Simple  list  element  is  ";  }  !bundle  agent  stats_iteration  {     vars:       "stats"     slist  =>  {  "value",  "av",  "dev"  };       "monvars"     slist  =>  {  "rootprocs",  "otherprocs",  "diskfree",  "loadavg"  };           reports:         "mon.$(stats)_$(monvars)  is  $(mon.$(stats)_$(monvars))";  }

It’s not rocket sciencebundle  agent  ensure_nginx_configuration_file()  {  !#  The  files  section  deals  with  promising  things  about,  well,  files!  files:  !   #  We  only  want  to  do  the  following  if  we’re  on  a  Linux  box  indicated  by  the  ‘linux’  hard  class     linux::       #  This  is  the  file  we  want  to  promise  will  be  there       “/opt/local/etc/nginx.conf”  !       #  These  are  the  properties  about  the  promise         create   =>   “true;         source   =>   local_dcp(“/var/cfengine/inputs/templates/linux-­‐nginx.conf”);         perms   =>   mog(“755”,”root”,”root”);         handle   =>   “ensure_linux_nginx_conf_maintained”;         comment   =>   “Maintain  the  nginx.conf  file  for  Linux  machines”;         classes   =>   if_repaired(“nginx_file_changed”);  !#  The  reports  section  allows  us  to  output  messages  to  the  user  reports:  !   #  We  only  want  to  say  something  if  something  has  changed     nginx_file_changed::       “WARNING:  NGINX  configuration  file  has  been  changed!  Restart  required!”;  !}

Further Reading & Doingwww.cfengine.com - CFengine AS company site. Documentation and a good starting point.

www.daemondreams.co.uk - an updated blog site where I keep articles and notes of interest on cfengine.

www.cfengineers.net - a community focused site lead by a group of consultants in cfengineering.

www.watson-wilson.ca - a highly recommended cfengineering consultant.

www.normation.com - a commercial cfengineering company with a project called Rudder which is cool.

www.loicp.eu/blog - a cfengine centric blog exploring some newer functions in 3.5.

evolvethinking.com/evolve-thinkings-free-cfengine-library/ - good cfengine library to help you get started.

THANK YOU