Python and Bash:from os.system to Fabric

Radosław Ganczarek

● Linux fanboy (from Slackware to ArchLinux)● 9 months as devops (implementing scripts

for managing bup in Bash and build scripts in Python)

● Writing deployment scripts as a part of web developer job (Fabric, Bash)

● Running Bash workshops

About me

Agenda● Needs and applications● Subprocess● Beyond subprocess● Disk operations● From Bash to Python● Recapitulation

Needs and applications

BuildbotBuildoutAnsibleSaltSconsWAF

Build systems

System toolsPortage (Gentoo)YumWicdUbuntu Software CenterOpenStackImage Packaging SystemSupervisor

Bash replacementipython --profile=pysh%<command>!<command>dirlist = !lsdirlist.grep(‘^a.*’)

Subprocess

Before 2.4:os.systemos.spawn*os.popen*popen2.*commands.*

Subprocess2.4 and after:subprocess

Replacementsos.systemos.spawnlpos.popencommands.getoutput

subprocess.callsubprocess.Popensubprocess.Popensubprocess.check_output

Beyond subprocess

Everybody likes to proxy subprocess!

shfrom sh import sort, du, glob, ifconfig, ls, cat, sudo

print sort(du(glob("*"), "-sb"), "-rn")

print ifconfig()

ls(_out='myfiles')print cat('myfiles')

ls = ls.bake('ls -Al')

cishimport cish, os

cish.rm("build")cish.default.pip("install", "nose")cish.default.python("setup.py", "build")cish.default.nosetests()

env = cish.from_interpreter("path/to/python")env.python("setup.py", "build")env.nosetests()

env = cish.from_config()[os.environ["PYTHON_VERSION"]]env.python("setup.py", "build")

venv = cish.default.virtualenv("optional/location")venv.pip("install", "package_to_install_inside_virtualenv")

popenfrom popen import Sh

for line in Sh('du', '-sh', '~/') | 'head -n 10' | 'sort -n':print('GOT', line)

Sh('ls', '-la', '~') > '~/listing'

Sh('ls', '-la', '~') | 'sort -c' | ['uniq', '-c'] | 'tail' | Sh('wc') >\ '~/listing'

cmd = Sh('ls') | 'grep polka'print cmd.returncode

print Sh.pipe(['this is a \nmulti line\nstring.']) | 'wc'

invokefrom invoke import task, run

@taskdef find_files(directory, pattern): print run( 'find {dir} -iname "{pat}"'.format( dir=directory, pat=pattern ) )

$ invoke find_files . 'C*'

FabricFabfileEnvironmentTasksRemote vs local operationsRun on all hostsParallel execution

File operations

Library vs Bash vs EzPython:fnmatchshutil.abspathshutil.copy*shutil.moveos.chdiros.remove*...

Bash:matching with maskpwdcpmvcdrm...

Ez:regexez.pwdez.cpez.mvez.cdez.rm...

From Bash to Python

>>> list(braceexpand('item{1..3}'))['item1', 'item2', 'item3']>>> list(braceexpand('{a..c}'))['a', 'b', 'c']>>> list(braceexpand('index.html{,.backup}'))['index.html', 'index.html.backup']>>> list(braceexpand('python{2.{5..7},3.{2,3}}'))['python2.5', 'python2.6', 'python2.7', 'python3.2', 'python3.3']>>> list(braceexpand('{07..10}'))['07', '08', '09', '10']>>> list(braceexpand('{a..g..2}'))['a', 'c', 'e', 'g']>>> list(braceexpand('{4..1}'))['4', '3', '2', '1']

braceexpand

genpipelinefrom genpipeline import pipefilter, iter_source

@pipefilterdef double(target): while True: item = (yield) target.send(item*2)

@pipefilterdef printer(): while True: item = (yield) print(item)

iter_source(range(10)) | (double() | printer())

Bash is cool

Python cooler

Use them both

Recapitulation

Questions?

Radosław GanczarekSenior Python Developer @ STX Next

Twitter: @dreamwalker_inEmail: radoslaw@ganczarek.in

Thank you!