Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software...
Transcript of Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software...
![Page 1: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/1.jpg)
Iago López Galeiras
Diary of a developer
https://goo.gl/Kkq7Vx LinuxCon North America - August 2016
![Page 2: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/2.jpg)
Iago López Galeiras
∘ Living in Berlin∘ Kinvolk co-founder∘ Worked together with CoreOS on rkt the last 20 months∘ A rkt maintainer
github: https://github.com/iaguistwitter: @iaguis
![Page 3: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/3.jpg)
Berlin-based software company building foundational Linux technologies
Some examples of what we work on...
OSTreegit for operating system binaries Weavescope
![Page 4: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/4.jpg)
Find out more about us…
Blog: https://kinvolk.io/blog
Github: https://github.com/kinvolk
Twitter: https://twitter.com/kinvolkio
Email: [email protected]
![Page 5: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/5.jpg)
PlanWhat is rkt?What tasks involve working on rkt?How do we develop and run tests?How do we work with other communities?How do we stay current?
![Page 6: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/6.jpg)
Warning!This section includes stolen slides!
![Page 7: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/7.jpg)
a modern, secure, composable container runtime
![Page 8: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/8.jpg)
rktsimple CLI toolgolang + Linuxself-contained
init system/distro agnostic
![Page 9: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/9.jpg)
an implementation of appc(image format, execution environment)
![Page 10: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/10.jpg)
a standard application containeropen specificationassociated tooling
![Page 11: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/11.jpg)
github.com/appc/spec ("appc spec")github.com/appc/acbuildgithub.com/appc/docker2acigithub.com/appc containernetworking/cnigithub.com/appc/...
![Page 12: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/12.jpg)
appc spec in a nutshell
- Image Format (ACI)- what does an application consist of?
- Image Discovery- how can an image be located?
- Pods- how can applications be grouped and run?
- Executor (runtime)- what does the execution environment look like?
![Page 13: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/13.jpg)
simple CLI toolno (mandatory) daemon:
apps run directly under spawning processswappable stage1s: nspawn, KVM, fly
![Page 14: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/14.jpg)
How did rkt start
∘ Not your typical open-source project∘ Well known from the start∘ Bay Area startup environment
∘ Hacker News∘ Hype∘ Startup wars∘ Container wars∘ Not what we’re used to in Berlin
![Page 15: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/15.jpg)
How do we communicate?
∘ Distributed team∘ People in San Francisco and in Berlin
∘ CoreOS team in SF∘ Kinvolk & CoreOS team in Berlin
∘ Not very easy at the beginning- Needed to learn a new project & tech- Time difference
![Page 16: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/16.jpg)
Communication tools
∘ GitHub∘ Slack (internal)∘ IRC (external) #rkt and #rkt-dev @ Freenode∘ rkt-dev mailing list∘ Internal syncs on Hangouts∘ Community syncs on Zoom (before May Hangouts On Air)
![Page 17: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/17.jpg)
Developer setup
∘ My setup:∘ Arch Linux∘ Golang∘ Vim∘ i3wm∘ Chromium∘ gnome-terminal
![Page 18: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/18.jpg)
MondayMorning: Slept
Afternoon: SleptEvening: Watched YouTube
Night: Filed PR!
Diary of a rkt developer
![Page 19: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/19.jpg)
Roles
∘ Developer∘ Maintainer
∘ Tester∘ Release manager
![Page 20: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/20.jpg)
MondayMorning: Wrote code
Afternoon: Wrote testsEvening: Wrote docs
Night: Helped new contributors
Diary of a rkt developer
![Page 21: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/21.jpg)
TuesdayMorning: Filed/fixed issues on projects rkt depends onAfternoon: Triaged issues
Evening: Cut a releaseNight: Organized community
sync
Diary of a rkt maintainer
![Page 22: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/22.jpg)
TuesdayMorning: Gave talk about rkt
Afternoon: SleptEvening: Slept
Night: Slept
Diary of a rkt maintainer
![Page 23: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/23.jpg)
Feature/bugfix workflow
∘ Somebody files an issue asking for a feature or complaining about a bug
∘ We tag and prioritize
![Page 24: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/24.jpg)
Feature workflow (1/5)
∘ Then somebody makes a PR for it
![Page 25: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/25.jpg)
∘ We review∘ General feedback about the PR∘ Code review. Ask for a test. Ask for doc changes
Feature workflow (2/5)
![Page 26: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/26.jpg)
Feature workflow (3/5)
∘ The developer force-pushes to the branch∘ Writes an “Updated” message∘ We repeat this n times. Until...
![Page 27: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/27.jpg)
Feature workflow (4/5)
∘ A maintainer writes LGTM (or LFAD :P)
![Page 28: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/28.jpg)
Feature workflow (5/5)
∘ We wait for CI to finish and…∘ We merge!
![Page 29: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/29.jpg)
∘ Unit testing on Travis∘ Works fine∘ rkt is complicated software
∘ Relies a lot on other software∘ We needed functional testing
You mentioned CI...
![Page 30: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/30.jpg)
∘ rkt is a special kind of software∘ Interacts a lot with the host and requires a lot of privileges on it∘ We didn’t want to have to create our own CI server∘ @alban did an initial analysis of CI services:
∘ https://github.com/coreos/rkt/issues/600#issuecomment-87655911
Functional testing
![Page 31: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/31.jpg)
∘ Travis: CAP_SYS_ADMIN ✗∘ Drone: Changing mount options ✗∘ Codeship: root ✗∘ Semaphore: Changing mount options ✗∘ Wercker: Changing mount options ✗∘ CircleCI: mknod ✗
CI service analysis
![Page 32: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/32.jpg)
∘ Semaphore runs tests on containers∘ But they gave us access to a beta image that uses VMs instead∘ After using that, we had all the privileges we needed to test rkt! Yay!!
Nothing works, but...
![Page 33: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/33.jpg)
∘ Semaphore has∘ An old Ubuntu (14.04 LTS v1503)∘ An old kernel (3.13)
∘ No userns∘ No overlay fs∘ …
∘ Plus we want to test other distros
All was fine except
![Page 34: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/34.jpg)
∘ Bite the bullet and set up our own CI∘ Jenkins + AWS plugin∘ Annoying UI∘ Complicated to set up∘ But we can test on arbitrary distros with arbitrary kernels
Enter Jenkins
![Page 35: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/35.jpg)
∘ Periodically test rkt∘ With lots of configurations
∘ Test each PR∘ More limited testing
∘ Distros∘ Fedora 22-rawhide∘ Debian testing∘ More to come...
Jenkins
![Page 36: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/36.jpg)
∘ Fighting test flakes∘ Make periodic test really pass
∘ Still some problems with certain rkt configs∘ Test KVM backend
∘ Not possible on AWS∘ Still use Semaphore for that∘ Some flaky tests
CI TODO
![Page 37: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/37.jpg)
∘ We depend/rely on several projects∘ We need to communicate our needs∘ Sometimes even implement them∘ Here’re some of the projects we collaborate with
Work with related projects
![Page 38: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/38.jpg)
∘ rkt uses and interfaces with∘ systemd-nspawn∘ systemd (PID 1)
∘ We use ∘ The mailing list∘ GitHub
∘ We talk to Lennart∘ he’s in Berlin too
systemd
![Page 39: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/39.jpg)
systemd.conf 2016Sept. 28th - Oct. 1st @ betahaus in Berlin
https://conf.systemd.io
Day 1: Hands on workshop session
Day 2 & 3: Presentations
Day 4: Hackfest
![Page 40: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/40.jpg)
∘ Some examples of issues we fixed:∘ Propagate exit status∘ notify-ready∘ InaccessiblePaths=∘ Cgroup issues∘ ...
systemd
![Page 41: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/41.jpg)
∘ Being a container runtime means∘ We use some of the latest features in Linux∘ Some are not so mature
∘ We use the mailing list of course∘ Examples:
∘ CONFIG_PROC_CHILDREN∘ cpuset issue∘ Use overlay fs and find some issues∘ Test cgroup v2
Linux kernel
![Page 42: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/42.jpg)
∘ We depend on a lot of go libraries∘ Go is hip -> GitHub∘ Some examples
∘ ql database∘ gexpect∘ docker2aci
go libraries
![Page 43: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/43.jpg)
∘ We want to help packagers to get rkt everywhere∘ Fedora… and eventually CentOS/RHEL∘ Debian∘ Arch Linux
Linux distributions
![Page 44: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/44.jpg)
∘ Abstraction over containers in clusters∘ Became major driving force of rkt∘ Pushing container runtime agnostic∘ Needed several new features in rkt
∘ Mostly rktnetes team interacting with k8s∘ They ask, we provide :)
Kubernetes
![Page 45: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/45.jpg)
∘ CoreOS∘ From the beginning :)
∘ Intel∘ KVM stage1∘ Communication: IRC and community syncs
∘ Blablacar∘ Issues and conferences
Work with other companies
![Page 46: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/46.jpg)
∘ We use a lot of cutting-edge technologies∘ How do we stay current?∘ Read∘ Fiddle∘ Read
How do we stay current
![Page 47: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/47.jpg)
- LWN- Great in-depth articles- All the new features in the kernel
- LKML- When LWN is not enough
- We analyze new features in systemd to see how we can benefit- Conferences- Twitter :)
How do we stay current
![Page 48: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/48.jpg)
That’s about it...
![Page 49: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/49.jpg)
[go to the next tab]
Long form blog story coming within the next couple days.
Twitter: @kinvolkio
Timeline & blog
![Page 50: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/50.jpg)
Get involvedGithub: https://github.com/coreos/rkt
Mailing list: https://groups.google.com/forum/#!forum/rkt-dev
#rkt-dev @ Freenode
Thanks!
![Page 51: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/51.jpg)
Find out more about us…
Blog: http://kinvolk.io/blog
Github: https://github.com/kinvolk
Twitter: https://twitter.com/kinvolkio
Email: [email protected]
![Page 52: Diary of a developer - events.static.linuxfound.org · ∘rkt is a special kind of software ∘Interacts a lot with the host and requires a lot of privileges on it ∘We didn’t](https://reader034.fdocuments.in/reader034/viewer/2022042313/5edd8862ad6a402d6668a7f7/html5/thumbnails/52.jpg)
Questions?