Provenance Issues in Platform-as-a-Service Model of Cloud ... · Solum - PaaS/ ALM for OpenStack...
36
Provenance Issues in Platform-as-a-Service Model of Cloud Computing Devdatta Kulkarni [email protected] PhD, University of Minnesota Minneapolis Affiliations: Rackspace, UT Austin
Transcript of Provenance Issues in Platform-as-a-Service Model of Cloud ... · Solum - PaaS/ ALM for OpenStack...
Provenance Issues in
Platform-as-a-Service Model of
Cloud Computing
De
vd
att
a K
ulk
arn
idevdattakulk
arn
i@gm
ail.
com
PhD
, U
niv
ers
ity o
f M
innesota
Min
neapolis
Affili
ations: R
ackspace, U
T A
ustin
Agenda
De
fin
e P
rove
na
nce
in
Pa
aS
Dis
cu
ss P
rove
na
nce
Issu
es in
Pa
aS
Pre
se
nt
me
ch
an
ism
s t
o a
dd
ress t
he issu
es
PaaS
Syste
ms th
at
allo
w a
pp
lica
tio
n d
eve
lop
ers
to
de
plo
y
the
ir a
pp
lica
tio
ns t
o c
lou
d in
fra
str
uctu
re “
ea
sily
”W
ithout
havin
g t
o w
orr
y a
bout
pro
vis
ion
ing the infr
astr
uctu
re
layer
(serv
ers
, data
base,
etc
.)
Als
o k
no
wn
as a
pp
lica
tio
n life
-cycle
ma
na
ge
me
nt
(AL
M)
syste
ms
E.g
. H
ero
ku
, G
oo
gle
Ap
p E
ng
ine
, O
pe
nS
hift,
C
lou
dF
ou
nd
ry,
So
lum
Pro
venance
Info
rma
tio
n a
bo
ut
an
en
tity
th
at
he
lps w
ith
un
de
rsta
nd
ing
ho
w t
ha
t e
ntity
go
t to
a p
art
icu
lar
sta
te
Th
e “
en
titie
s”
tha
t w
e c
on
sid
er
are
-T
he p
latf
orm
its
elf
-A
pplic
atio
ns d
ep
loye
d b
y t
he p
latf
orm
Pro
venance E
xam
ple
s
Wh
en
de
ve
lop
ing
Pa
aS
, w
ha
t w
ere
th
e e
xa
ct se
t o
f co
mm
an
ds u
se
d to
in
sta
ll a p
art
icu
lar
libra
ry/to
ol?
Wh
en
a P
aa
Sis
de
plo
ye
d,
wh
at
are
th
e v
alu
es o
f th
e
co
nfig
ura
tio
n p
ara
me
ters
fo
r d
iffe
ren
t se
rvic
es?
Wh
en
an
ap
plic
atio
n is d
ep
loye
d b
y P
aa
S,
wh
at
is t
he
ve
rsio
n o
f D
ocke
ru
se
d t
o b
uild
ap
plic
atio
n
co
nta
ine
rs?
PaaS
and P
rovenance
Pa
aS
ma
na
ge
s c
om
ple
te life
-cycle
of a
n a
pp
lica
tio
n
Pro
ve
na
nce
is im
po
rta
nt
•F
or
PaaS
deve
lopers
and o
pera
tors
To e
nable
corr
ect desig
n a
nd o
pera
tion o
f th
e P
aaS
•F
or
applic
ation d
eve
lopers
To g
ain
insig
hts
into
applic
ation c
onstr
uction p
rocess
To g
ain
confidence in the w
ork
ing o
f a P
aaS
Solu
m -
PaaS
/ A
LM
for
OpenS
tack
•S
up
port
s d
ep
loyin
g a
pp
licatio
ns s
tart
ing fro
m the s
ourc
e
code
•C
usto
m L
angu
ag
e P
ack m
echanis
mJava, P
yth
on, N
odeJS
, R
uby, 9
•A
pp
lication
s a
re c
onstr
ucte
d a
s D
ocker
conta
iners
•U
ses O
pen
Sta
ck s
erv
ices
-Keysto
ne for
auth
entication
-Gla
nce a
nd S
wift to
sto
re c
onta
iner
images for
lan
guage r
untim
es,
applic
ation c
onta
iner
images, lo
gs
-Heat and N
ova to d
eplo
y a
pplic
ation c
onta
iners
•A
llow
s o
ptiona
lly r
unn
ing o
f te
sts
Continuous inte
gra
tion for
applic
ations
•Is
inte
gra
ted w
ith G
ithub
Applic
ation d
eplo
ym
ent can b
e triggere
d b
y G
ithub
we
bhooks
Agenda
De
fin
e P
rove
na
nce
in
Pa
aS
Pro
ve
na
nce
Issu
es in
Pa
aS
–PaaSdevelopment
–PaaSbuilding
–P
aa
Sdep
loym
ent
–A
pp
lication
s o
n P
aaS
Me
ch
an
ism
s t
o a
dd
ress t
he issu
es
PaaS
develo
pm
ent
So
lum
exp
eri
en
ce
–S
evera
l soft
ware
sre
quire
d,
such a
s D
ocker,
Docker
regis
try,
Tom
cat, S
wift, G
lance,
Keysto
ne,
Nova
–In
sta
llation o
f a s
/wbefo
re it
can b
e u
sed w
as a
trial and
err
or
pro
cess
Typ
ica
l co
mm
an
d lin
e c
on
tain
s lo
t o
f co
mm
an
ds o
f
diffe
ren
t kin
ds
Navig
atio
nal com
man
ds (
cd, pusd),
Lis
ting/v
iew
ing
com
mands (
ls, le
ss),
editin
g c
om
mands (
vi, e
macs)
Pro
ve
nance f
or
Pa
aS
de
velo
pm
ent
•O
nce
a s
/wh
as b
ee
n s
uccessfu
lly insta
lled
, w
e
do
n’t w
an
t to
re
pe
at
the
pro
ce
ss f
rom
sta
rt a
ga
in
•Is
it
po
ssib
le t
o automatically
ge
ne
rate
th
e lis
t o
f
co
mm
an
ds r
eq
uir
ed
to
in
sta
ll a
pa
rtic
ula
r s/w
?
•W
e n
ee
de
d provenance
of so
ftw
are
in
sta
lls
•T
his
re
qu
ire
me
nt
is n
ot
co
nfin
ed
to
de
ve
lop
me
nt
of
Pa
aS
bu
t a
rise
s in
an
y d
eve
lop
me
nt
sce
na
rio
th
at
ne
ed
s t
o in
sta
ll a
nd
use
ne
w s
/w
PaaS
build
ing
So
lum
exp
eri
en
ce
–O
penS
tack s
erv
ices p
rogre
ss in
depe
nd
ently
–C
han
ges in
depen
de
nt
serv
ices m
ay c
ause S
olu
m t
o
sto
p b
uild
ing
We
wa
nt
to u
nb
lock S
olu
m b
uild
s b
y p
inn
ing
to
an
e
arl
ier
co
mm
it o
f a s
erv
ice
Fig
uri
ng
ou
t co
mm
it(s
) b
rea
kin
g S
olu
m in
de
pe
nd
en
t se
rvic
es is a
ma
nu
al a
nd
te
dio
us p
roce
ss
Pro
venance for
PaaS
build
s
Re
aliz
atio
n -
Every
tim
e S
olu
m is s
uccessfu
lly b
uilt
, m
ain
tain
info
rma
tion
abo
ut th
e c
om
mits o
f th
e d
ep
en
dent
serv
ice
s u
sed (
sim
ilar
to “
Glo
ba
l re
sto
re p
oin
ts”
of A
pp-B
isect)
•T
his
info
rmation w
ill b
e u
sefu
l w
hen S
olu
m b
uild
s f
ail
in t
he
futu
re
•F
indin
g c
ulp
rit
com
mit in a
depe
ndent
serv
ice c
an s
tart
fro
m
the last
know
n g
ood c
om
mit w
ith w
hic
h S
olu
m w
as
successfu
lly b
uilt
•W
e n
eede
d provenance o
f successfu
l S
olu
m b
uild
s
git
bis
ect
for
usin
g d
epen
dent
serv
ices
Agenda
De
fin
e P
rove
na
nce
in
Pa
aS
Pro
ve
na
nce
Issu
es in
Pa
aS
Me
ch
an
ism
s t
o a
dd
ress t
he issu
es
–PaaSdevelopment
•Command List Provenance
–PaaSbuilding
•Commit tracking
•Merge tracking
–P
aa
Sdep
loym
ent
•C
onfigura
tion p
ara
mete
r tr
ackin
g
–A
pp
lication
s o
n P
aaS
•O
bje
ct m
odel and A
PI
Com
mand L
ist P
rovenance
Pro
ble
mG
iven s
he
ll his
tory
fin
d the lis
t of com
mands t
hat
repre
sent
pro
ven
ance o
f a s
oftw
are
’s insta
llation
•So
lutio
n o
utlin
e–C
reate
can
did
ate
lis
t of com
mand
s
–T
ry the c
andid
ate
lis
t in
an a
uto
mate
d m
ann
er
–V
erify
that th
e c
and
ida
te lis
t le
ads to s
oftw
are
’s
insta
llatio
n (
the s
oftw
are
’s p
rove
nance)
•Ch
alle
ng
es
–H
ow
to d
eal w
ith lon
g s
he
ll h
isto
ries?
–H
ow
to d
ete
rmin
e t
hat
a c
andid
ate
lis
t of
com
mands
repre
se
nts
the p
roven
ance o
f in
sta
llin
g a
softw
are
?
•T
o a
ddre
ss long s
he
ll his
tory
–D
efine A
nchor
Poin
t (A
P)
com
mands
–T
hese a
re c
om
mands w
hic
h p
rovid
e s
tart
ing a
nd s
toppin
g p
oin
ts
within
the s
hell
his
tory
E.g
.: a
pt-
get update
on U
buntu
•T
o a
ddre
ss a
uto
mate
d trial an
d v
erification o
f com
mand lis
t–
Use c
apabili
ties o
ffere
d b
y Docker
–B
uild
a Dockercontainer
with c
andid
ate
com
mand lis
t
–V
erify
the lis
t usin
g a
verification script
•V
erificatio
n s
cript
Defines c
hecks to v
erify
that th
e s
oftw
are
was c
orr
ectly insta
lled
Com
mand L
ist P
rovenance
•F
ea
sib
ility
stu
dy (
initia
l e
xp
eri
me
nta
tio
n)
–In
sta
lled D
ocker
and T
om
cat
on H
ost
–T
ried t
he c
om
mand lis
t pro
vena
nce a
ppro
ach t
o fin
d o
ut
pro
ven
ance f
or
both
•V
eri
fica
tio
n S
cri
pts
–F
or
Docker
Check o
utp
ut of “docker–v”
com
mand
–F
or
Tom
cat
Check p
resence o
f “webapps”
fold
er
at a k
now
n location
Com
mand L
ist P
rovenance
Com
mand h
isto
ries
Tom
cat
----
----
---
% :
% t
om
cat
% a
pt-
get
upd
ate
% a
pt-
get
insta
ll -y
tom
cat7
% c
url loca
lhost:
8080
Docker
----
----
---
% a
pt-
get
upgra
de
% a
pt-
get
upd
ate
% a
pt-
get
-y u
pgra
de
% u
nam
e-r
% w
hic
h w
get
% w
get
-qO
-htt
ps:/
/get.
docker.
com
/ | sh
% d
ocker
-v
Com
mand L
ist P
rovenance
Sam
ple
Observ
ation
Observ
atio
nD
ocker
build
may fail
if:
•C
om
man
d is a
not-
exis
tent
co
mm
and
Happened w
ith tomcat
•C
om
man
d is n
ot
insta
lled o
n t
he c
onta
iner
(an
y la
yer)
Happened w
ith curl
»W
as p
resent on
the h
ost but
its insta
llation w
as n
ot
part
of
the
candid
ate
com
mand lis
t
Re
aliz
atio
nB
efo
re inclu
din
g a
com
mand in Dockerfile
run it on the h
ost
•In
clu
de it
on
ly if
it r
an s
uccessfu
lly o
n t
he h
ost
Rulin
g o
ut in
clu
sio
n o
f tomcat
•In
clu
de insta
llation c
om
man
dE
nsuring curl is insta
lled o
n the c
onta
iner
Co
mm
and lis
t pro
v:
Observ
ation
s (
2/3
)
Ob
se
rva
tio
n
A c
om
mand m
ay n
eed d
iffe
rent fla
gs t
o e
xe
cute
on c
onta
iner
as c
om
pare
d t
o the h
ost
•H
appened w
ith c
ert
ific
ate
checkin
g b
y w
get
•H
ad to intr
oduce -
-no-c
heck-c
ert
ific
ate
fla
g o
n c
onta
iner
–w
get
-v -
-no-c
heck-c
ert
ific
ate
htt
ps:/
/get.
docker.
com
/
Re
aliz
atio
n
Ne
ed t
o fig
ure
out
appro
priate
set
of flags t
o u
se w
hen
inclu
din
g t
he c
om
mand in D
ockerf
ile
Co
mm
and lis
t pro
v:
Observ
ation
s (
3/3
)
Oth
er
ob
se
rva
tio
ns
–P
iped c
om
mands o
n h
ost
may n
eed t
o b
e s
plit
befo
re
inclu
din
g in
Dockerf
ile
–N
avig
atio
n c
om
mands (
cd/p
ushd/p
op
d)
nee
d to b
e
com
bin
ed w
ith o
ther
com
mands w
hen inclu
din
g in t
he
RU
N c
om
mand in D
ockerf
ile
–F
iles m
odifie
d o
n t
he h
ost
can b
e c
op
ied into
the
conta
iner
at appro
pri
ate
location
Com
mand lis
t pro
v: O
ther
Issues
Ho
w t
o h
an
dle
exte
rna
l d
ep
en
de
ncie
s a
nd
th
eir
ve
rsio
ns w
he
n t
he
s/w
wa
s in
sta
lled
on
th
e h
ost?
One idea is to d
ete
rmin
e v
ers
ion
s o
f exte
rnal packag
es w
hen
the s
oftw
are
was insta
lled o
n t
he h
ost
Possib
le to fin
d o
ut usin
g:
dpkg
–l
pip
fre
eze
Serv
ice d
ep
enden
cy tra
ckin
g
Pro
ble
mG
iven a
faili
ng S
olu
m b
uild
ho
w to d
ete
rmin
e w
hic
h s
erv
ice
and w
hic
h c
om
mit o
f it is the c
ause o
f th
e b
uild
failu
re
So
lutio
ns
–T
rack d
epe
nde
nt
serv
ices a
nd t
heir c
om
mits
–T
rack d
epe
nde
nt
serv
ices a
nd t
heir “
merg
e-t
o-m
aste
r”events
Com
mits tra
ckin
g
•T
rack c
om
mits o
f d
ep
en
de
nt
se
rvic
es t
ha
t le
ad
to
su
cce
ssfu
l S
olu
m b
uild
s–
Last
successfu
l build
:•<Keystone=a, Tempest=1, Glance=A>
•S
up
po
se
So
lum
bu
ild f
ails
an
d t
he
curr
en
t co
mm
its
of
de
pe
nd
en
t se
rvic
es a
re:
–<Keystone=c, Tempest=3, Glance=B>
•H
ow
to
de
term
ine
wh
ich
se
rvic
e a
nd
wh
ich
co
mm
it
is c
au
sin
g S
olu
m b
uild
to
fa
il?
Com
mit d
epende
ncy g
raph
c b a
3 2 1
C B A
Ke
ysto
ne
Tem
pest
Gla
nce
depe
ndsO
n
Fin
din
g b
reakin
g c
om
mit
•S
ele
ct
a d
ep
en
de
nt
se
rvic
e S
•C
he
ck t
he
la
test co
mm
it t
o s
ee
if
So
lum
bu
ilds
su
cce
ssfu
lly
•If
no
t, r
em
ove
th
e c
om
mit f
rom
co
nsid
era
tio
n
•R
em
ove
all
the
com
mits f
rom
oth
er
se
rvic
es t
ha
t fo
rm a
transitive closure o
f th
e needed-by
gra
ph
(e
dg
es r
eve
rse
d f
rom
th
e d
ep
en
de
ncy g
rap
h)
Com
mit d
epende
ncy g
raph
c b a
3 2 1
C B A
Ke
ysto
ne
Tem
pest
Gla
nce
Cand
idate
com
mit:<
Keysto
ne
=c, T
em
pest=
3,
Gla
nce=
B>
Cand
idate
com
mit:<
Keysto
ne
=b, T
em
pest=
3,
Gla
nce=
A>
depe
ndsO
n
Questions
•H
ow
to
bu
ild t
he
de
pe
nd
en
cy g
rap
h?
–P
ossib
le if
each s
erv
ice m
ain
tain
s p
rovena
nce f
or
its
successfu
l bu
ilds
–S
erv
ice d
epen
de
ncie
s form
a D
AG
•In
wh
ich
ord
er
de
pe
nd
en
t se
rvic
es s
ho
uld
be
tr
ied
?
Ke
ysto
ne
Gla
nce
Nova
Heat
Merg
e tra
ckin
g
Ide
aW
henever
ne
w c
ode m
erg
es t
o m
aste
r in
depe
nd
ent
serv
ices,
proactively
check t
hat
it d
oes n
ot aff
ect S
olu
m
Op
en
Sta
ck C
I S
yste
ms
–Z
uu
l•
Runs tests
•P
roje
ct definitio
n–
wh
ich t
ests
to r
un
–G
err
it•
Code g
ating a
nd r
evie
ws
Merg
e tra
ckin
g
•E
nh
an
ce
pro
ject
de
fin
itio
n in
Zu
ulw
ith
a Trigger
eve
nt
an
d lis
t o
f using p
roje
cts
•O
n m
erg
e-t
o-m
aste
r a
dd
a “
rech
eck n
ob
ug
”co
mm
en
t to
ou
tsta
nd
ing
pa
tch
es f
or Used_by
pro
jects
in
Ge
rrit
–T
his
com
ment
trig
gers
a C
I ru
n o
n t
he p
roje
ct
Barbican:
Triggers:
OnMergeToMaster:
Used_by:
Solum, Murano, Mistral
Questions
•W
hat m
odific
ations a
re n
eeded to Z
uulto
enable
merg
e-t
o-m
aste
r event genera
tion?
•W
hat if n
o o
uts
tandin
g p
atc
hes a
re p
resent
for
a p
roje
ct?
–S
ho
uld
Z
uu
lg
en
era
te a
pa
tch
?
–W
ha
t w
ill b
e t
he
na
ture
of
such
a p
atc
h?
Pro
ve
nance o
f P
aa
Sdeplo
ym
en
t
So
lum
exp
eri
en
ce
–E
ach O
pen
Sta
ck s
erv
ice h
as larg
e n
um
ber
of
config
ura
tion p
ara
mete
rs
–N
ot
sett
ing c
orr
ect
para
mete
rs in d
ep
en
de
nt serv
ices t
o
appro
pri
ate
valu
es c
au
sed S
olu
m t
o n
ot deplo
y
Pro
ble
mN
eed t
o k
now
whic
h p
ara
mete
rs in d
epe
nd
ent
serv
ices a
re
critica
l fo
r S
olu
m d
ep
loym
ent
Po
ten
tia
l so
lutio
nT
rack p
ara
mete
rs a
nd their v
alu
es for
depen
dent
serv
ices
Infr
astr
uctu
re C
onfigura
tion tra
ckin
g
Pro
ble
m
On
de
ve
lop
me
nt
ve
rsio
n o
f O
pe
nS
tack (
de
vsta
ck),
co
nfig
pa
ram
ete
rs a
nd
va
lue
s o
f d
ep
en
de
nt
se
rvic
es
ne
ed
to
be
tra
cke
d
Solu
tion
Ve
rsio
n c
on
tro
l
Pro
ve
nance o
f applic
ation
on P
aaS
•W
hic
h r
evis
ion
of a
pp
lica
tio
n c
od
e u
se
d t
o c
rea
te
ap
plic
atio
n c
on
tain
er
in a
pa
rtic
ula
r d
ep
loym
en
t?
•W
hic
h r
evis
ion
s o
f syste
m lib
rari
es a
nd
so
ftw
are
u
se
d f
or
ap
plic
atio
n c
on
str
uctio
n,
bu
ildin
g,
an
d
de
plo
ym
en
t?–
Revis
ion o
f D
ocker
used
–R
evis
ion o
f H
eat
used
–R
evis
ion o
f G
lance/S
wift used
Applic
ation P
rovenance A
PI
Pro
ble
mN
eed t
o m
ain
tain
info
rmation a
bout
every
app
licatio
n b
uild
and
dep
loym
en
t
–A
pp
lication
-leve
l in
form
ation
•S
ou
rce
co
de
co
mm
it
•T
est
an
d r
un
co
mm
an
d u
se
d
–In
frastr
uctu
re-leve
l in
form
ation
•V
ers
ion
of
Do
cke
ru
se
d
•V
ers
ion
of
He
at
use
d
So
lutio
n–
Mechan
ism
s to tra
ck this
info
rmation
–A
PI to
extr
act
it
PaaS
develo
pm
ent
•O
ther
aspects
–O
nlin
e t
uto
ria
ls u
se
d
–C
od
e s
am
ple
s u
se
d
–S
tacko
verf
low
links r
efe
rre
d t
o
•T
rackin
g p
rovenance o
f pla
tform
code
develo
pm
ent
–N
ot
co
nsid
ere
d
Conclu
sio
n
•P
rese
nte
d issu
es a
risin
g in
de
ve
lop
ing
an
d
bu
ildin
g P
aa
S
•A
rgu
ed
th
at provenance c
an
be
use
d to
ad
dre
ss
the
se
issu
es
–Id
entified t
he n
ee
de
d p
roven
ance info
rmation
–P
resente
d m
echan
ism
s to c
olle
ct and u
se t
his
in
form
atio
n
•M
ore
de
tails
ava
ilab
le in
th
e p
ap
er
