Joe Damato
-
Upload
ontico -
Category
Technology
-
view
329 -
download
2
description
Transcript of Joe Damato
![Page 1: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/1.jpg)
scaling compiled applications Joe Damato
@joedamato
![Page 2: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/2.jpg)
about me • Joe Damato
• San Francisco, CA
• i like:
• systems
programming
• debugging
• performance
analysis
![Page 3: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/3.jpg)
@joedamato
![Page 4: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/4.jpg)
timetobleed.com
![Page 5: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/5.jpg)
sassanosystems.c
om
![Page 6: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/6.jpg)
![Page 7: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/7.jpg)
![Page 8: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/8.jpg)
![Page 9: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/9.jpg)
![Page 10: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/10.jpg)
![Page 11: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/11.jpg)
![Page 12: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/12.jpg)
![Page 13: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/13.jpg)
![Page 14: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/14.jpg)
![Page 15: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/15.jpg)
![Page 16: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/16.jpg)
![Page 17: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/17.jpg)
![Page 19: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/19.jpg)
![Page 20: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/20.jpg)
![Page 21: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/21.jpg)
Repeatable builds • the most important thing for compiled software
• set up a build server
• with jenkins (or another type of software)
• backup your built objects
• copying them to s3 is not a horrible idea
• regardless of source control system or branching strategy,
ensure you can always rebuild any version of your
software
![Page 22: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/22.jpg)
![Page 23: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/23.jpg)
Jenkins
problems • git plugin didn’t work on windows (maybe
fixed?)
• branch building is painful, jenkins API can
help
• getting windows build agent working is
painful
![Page 24: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/24.jpg)
![Page 25: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/25.jpg)
![Page 26: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/26.jpg)
why?
почему?
![Page 27: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/27.jpg)
![Page 28: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/28.jpg)
![Page 29: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/29.jpg)
tools for... • RPMs
• DEBs
• Everything else
• windows installers (MSIs)
• other linux/unix like OS’s
• etc
![Page 30: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/30.jpg)
chroot ??
![Page 31: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/31.jpg)
chroot: • an operation that changes the apparent root directory
for the current running process [...].
• A program that is run in such a modified environment
cannot name (and therefore normally not access) files
outside the designated directory tree.
• (from wikipedia)
![Page 32: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/32.jpg)
![Page 33: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/33.jpg)
![Page 34: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/34.jpg)
![Page 35: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/35.jpg)
![Page 38: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/38.jpg)
Everything else
• KVM
• Amazon EC2
• other virtualization
![Page 39: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/39.jpg)
KVM EC2
![Page 40: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/40.jpg)
KVM
• Create a base image on disk
• Clone base image
• Boot the cloned image
• Do the build and copy built object out.
• Delete cloned image when done
• Base image is still pristine and can be reused.
![Page 41: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/41.jpg)
![Page 42: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/42.jpg)
![Page 43: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/43.jpg)
Create builds in cleanroom • Avoid contaminating builds with artifacts from previous
builds.
• chroots help
• use mock or pbuilder for RPMs and DEBs
• KVM, EC2, or equivalent for everything else
• Always create pristine base images and do builds in a
copy.
• Use SSDs
![Page 44: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/44.jpg)
![Page 45: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/45.jpg)
Tool problems • git-buildpackage can’t set changelog distribution field
• signing key setup is really painful (especially for
RPMs)
• deb naming scheme for packages is quite painful
• all tools are undocumented and very hard to actually
use
• recent versions of KVM provided by ubuntu fail to
boot VMs sometimes
![Page 46: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/46.jpg)
![Page 47: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/47.jpg)
![Page 48: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/48.jpg)
two types of
linking....
• dynamic linking
• static linking
![Page 49: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/49.jpg)
static linking
• calls to functions in library are resolved at compile
time.
• code from the library is copied into the resulting
binary.
![Page 50: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/50.jpg)
dynamic
linking • calls to functions are resolved at
runtime.
• code for a library lives in it’s own
object:
• libblah.so.4
• libblah.dll
![Page 51: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/51.jpg)
![Page 52: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/52.jpg)
http://www.akkadia.org/drepper/no_static_linking.html
![Page 53: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/53.jpg)
![Page 54: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/54.jpg)
why?
почему?
![Page 55: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/55.jpg)
![Page 56: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/56.jpg)
![Page 57: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/57.jpg)
![Page 58: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/58.jpg)
static
linking • figure out which libraries your app needs
• pick a supported major release, if possible
• build and package this library
• link it statically against your binary during build
• you now have fewer stones to turn over when
debugging
![Page 59: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/59.jpg)
![Page 60: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/60.jpg)
static linking • you will need to track your upstream deps
• you will probably want to package your upstream
deps
• you can then point your chroots and build envs at
private deb, rpm, etc repos in your internal
infrastructure
• merging upstream changes in becomes its own
project
![Page 61: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/61.jpg)
![Page 62: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/62.jpg)
![Page 63: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/63.jpg)
![Page 64: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/64.jpg)
![Page 65: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/65.jpg)
why?
почему?
![Page 66: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/66.jpg)
![Page 67: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/67.jpg)
![Page 68: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/68.jpg)
![Page 69: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/69.jpg)
![Page 70: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/70.jpg)
![Page 71: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/71.jpg)
![Page 72: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/72.jpg)
![Page 73: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/73.jpg)
![Page 74: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/74.jpg)
Use files • src/
•redhat5/
•some_internal_thing.c
•ubuntu1204/
•some_internal_thing.c
•debian6/
•some_internal_thing.c
![Page 75: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/75.jpg)
Use the build system
Determine which file
to build at compile
time.
![Page 76: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/76.jpg)
![Page 77: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/77.jpg)
![Page 78: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/78.jpg)
Use
modules
• break up ifdef soup into separate files
• use the build system to compile the right file at
build time
• this seems obvious but many C libraries and
programs are full of crazy ifdef soup.
![Page 79: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/79.jpg)
![Page 80: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/80.jpg)
Use modules
• very easy to fall down a rabbit hole breaking things
apart
• can make the build process more complicated and
harder to debug
![Page 81: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/81.jpg)
![Page 82: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/82.jpg)
![Page 83: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/83.jpg)
why?
почему?
![Page 84: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/84.jpg)
![Page 85: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/85.jpg)
Capture debug
symbols
• DEB and RPM can both output debug packages
that contain debug symbols.
• output these packages.
• store these and make backups.
• (or just don’t strip your binary)
![Page 86: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/86.jpg)
![Page 87: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/87.jpg)
Use google-
coredumper • you can use google-coredumper to catch
segfaults, bus errors, and other bad things.
• you can output a coredump when this
happens.
• you can use this coredump and your debug
symbols to figure out what is going on.
![Page 88: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/88.jpg)
Plan for
failure • Have a backup plan
• Capture debug symbols during your automated
build process.
• Store them somewhere safe (and make
backups).
• Capture coredumps (if possible).
• Use coredumps and debug symbols to figure
out what happened.
![Page 89: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/89.jpg)
![Page 90: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/90.jpg)
Plan for
failure • can significantly increase complexity
• google coredumper can’t help if your kernel is
buggy
• some linux distributions don’t allow ptrace
• google coredumper only supports linux
![Page 91: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/91.jpg)
![Page 92: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/92.jpg)
![Page 93: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/93.jpg)
![Page 94: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/94.jpg)
![Page 95: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/95.jpg)
![Page 97: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/97.jpg)
![Page 98: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/98.jpg)
![Page 99: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/99.jpg)
Check things
like... • Is the binary actually statically linked?
• Does it get copied to the right path?
• Are the right config files autogenerated?
• Does the version string the program outputs match
the package version?
• ....
![Page 100: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/100.jpg)
You also need... correctness
testing.
![Page 101: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/101.jpg)
RSpec is useful for
this.
![Page 102: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/102.jpg)
Automated
Testing • It will be impossible to build and test every
change on every supported platform.
• Use your build server to do this for you.
• Test things like:
• installing/uninstalling the object
• object is actually statically linked
• correctness testing (RSpec can be useful)
![Page 103: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/103.jpg)
![Page 104: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/104.jpg)
Automated
testing • Fine line between not enough and too much
• Windows can be painful, but cygwin can help with
scripting
• Easy to forget about testing the actual package
install/removal
• Can be difficult to get working with branch builds
![Page 105: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/105.jpg)
To
summarize...
![Page 106: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/106.jpg)
![Page 107: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/107.jpg)
![Page 108: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/108.jpg)
![Page 109: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/109.jpg)
![Page 110: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/110.jpg)
![Page 111: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/111.jpg)
![Page 112: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/112.jpg)
![Page 113: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/113.jpg)
sassanosystems.c
om
@joedamato
![Page 114: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/114.jpg)
спасиб
о
![Page 115: Joe Damato](https://reader038.fdocuments.in/reader038/viewer/2022102616/559bbbd31a28ab971a8b469b/html5/thumbnails/115.jpg)