DevOps For Embedded Software Development
-
Upload
scotthal -
Category
Technology
-
view
704 -
download
2
description
Transcript of DevOps For Embedded Software Development
![Page 1: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/1.jpg)
DEVOPS FOR EMBEDDED SOFTWARE DEVELOPMENTScott Hallock
May 20, 2013
![Page 2: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/2.jpg)
Agenda• Introduction• What We Do• How We Do It• Strengths and Challenges
![Page 3: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/3.jpg)
Introduction
BAIT • Build, Automation, Integration, and Test• Continuous integration and automated release for Linux
software products• 3-5 Linux-based operating systems• Software releases nearly every day• Build Linux from source thousands of times per day• Support a worldwide Linux developer community of more than
1,000• More than 50 actively developed codelines• Integrate more than 200 code changes per day
![Page 4: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/4.jpg)
Objectives• High-level overview of embedded DevOps processes and
practices• Start an exchange of knowledge between embedded and
non-embedded DevOps communities• Interactivity
• Please interrupt to ask questions• Detailed, tangential discussions are welcome
![Page 5: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/5.jpg)
Embedded Development Is Different• Tools don’t evolve quickly
• Command line compilers• Everybody loves the preprocessor• printf debugging• Eclipse viewed with suspicion
• Product focus• Customers get the code, not just the experience• Fix it, ship it, forget it• Standard release: zip archive• Many similar codelines under active development simultaneously
![Page 6: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/6.jpg)
Continuous Integration
Review Build Deploy Test Integrate
![Page 7: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/7.jpg)
Continuous Integration Deployment
• Datacenter-quality hardware• Rack-mountable form factor• Stable, controlled operating system and management
software• Well-defined power and thermal characteristics
![Page 8: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/8.jpg)
Embedded Continuous Integration
Review Build Load Test Integrate
![Page 9: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/9.jpg)
Target Load Challenges
• Developer-quality hardware• Irregular form factor• Unstable loading software, boot code, operating system• Unknown power and thermal characteristics• Slow and unstable JTAG, USB loading interfaces
![Page 10: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/10.jpg)
Target Load Solutions• Single controlling server for each target hardware board• Physical access to production hardware
• Easily reconfigure physical racks to accommodate novel form factors
• Repair or replace unstable, malfunctioning target hardware
• Fault-tolerant software• Lots of retries!• Heuristics to infer target state based on observable outputs
• Change the target hardware design
![Page 11: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/11.jpg)
Time Constraints
Review Build Load Test Integrate
90 minutes
15 minutes
![Page 12: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/12.jpg)
Preverification
Build Load Test Integrate
90 minutes, many changes
Build
Review
60 minutes, one change
![Page 13: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/13.jpg)
Automated Release
Build Test Approve git push
![Page 14: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/14.jpg)
Code Change Control: git• http://git-scm.com• Distributed is great!
• Developers can work offline, at customer sites with no network access.
• Collaboration is easier. Everyone has complete history.
![Page 15: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/15.jpg)
git Problems• Everyone gets everything
• Do you check in a helper binary every time you change it?• Did you accidentally commit some trade secrets and then revert
them later?• Did your developers include an estimate of your customers’
intelligence in their commit message?• You create integration tags fifty times per day.• Your continuous integration system creates “temporary” integration
branches twenty-five times per day.
![Page 16: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/16.jpg)
git Problems• git trees become unusable over time. Are you garbage
collecting?• On your central repositories?• When you mirror your central repositories?• On your external release servers?• On your Continuous Integration build machines?
![Page 17: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/17.jpg)
git Problems• Cherry-picking
• I can just have the two changes I want for today’s release? Great!• Breaks all traceability, barring commit message kludges
![Page 18: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/18.jpg)
Codeline Definition: repo• http://source.android.com/source/using-repo.html• Allows for many git projects to be combined into a single
source tree and operated upon as a single entity.• Developers use it to download and sync code, most use
git commands for everything else.
<project name=“kernel/msm” path=“kernel” revision=“refs/heads/msm-3.4” /><project name=“kernel/lk” path=“bootable/bootloader/lk” revision=“refs/heads/master” />
![Page 19: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/19.jpg)
repo: Project Addition
Build Test Approve git push!!!
![Page 20: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/20.jpg)
repo: Branch Change
<project name=“kernel/msm” path=“kernel” revision=“refs/heads/msm-3.4” /><project name=“kernel/lk” path=“bootable/bootloader/lk” revision=“refs/heads/stable” />
<project name=“kernel/msm” path=“kernel” revision=“refs/heads/msm-3.4” /><project name=“kernel/lk” path=“bootable/bootloader/lk” revision=“refs/heads/master” />
![Page 21: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/21.jpg)
repo: Branch Change
Build Test Approve git push!!!
![Page 22: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/22.jpg)
repo: Branch Change• Merge?
• “master” branch contains dozens of bugs. You just merged them with your stable branch.
• Force push?• Now your customer doesn’t understand what happened to all the
changes they painstakingly integrated for the last month.
• Create new branch on release server?• Customer still has to re-integrate their changes.• At least the old branch is available for reference.
• Stop doing that!• How am I supposed to make my release?
![Page 23: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/23.jpg)
repo: Shared Branches
customer-2.xml
<project name=“kernel/msm” path=“kernel” revision=“refs/heads/msm-3.4” /><project name=“kernel/lk” path=“bootable/bootloader/lk” revision=“refs/heads/customer-2” />
customer-1.xml:
<project name=“kernel/msm” path=“kernel” revision=“refs/heads/msm-3.4” /><project name=“kernel/lk” path=“bootable/bootloader/lk” revision=“refs/heads/customer-1” />
![Page 24: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/24.jpg)
Gerrit: Code Review and More• https://code.google.com/p/gerrit/• Code review, access control, source code mirroring
![Page 25: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/25.jpg)
Gerrit: Code Review• States Tracked:
• Code Review (-2 - +2): Peer opinion of design and code quality• Verified (-1 - +1): Code works at runtime
• Roles• Author/Committer• Approver• Integrator
![Page 26: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/26.jpg)
Gerrit: Access Control• Most actions in Gerrit are access controlled
• Code visibility• Upload• Review• Merge
• Access control can be based on users, on local groups, or on LDAP groups.
• Access control is complex. Automation is recommended.
![Page 27: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/27.jpg)
Gerrit: Mirroring
Master
Slave Slave Slave
• Single master• Slaves can’t mirror to other slaves• Smart git push based mirroring
• Better than any mirroring scheme not based on application layer knowledge of git
• Errors not handled gracefully• Obvious errors are re-queued for later retries
![Page 28: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/28.jpg)
Code Aurora Forum• http://codeaurora.org
![Page 29: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/29.jpg)
Other Tools• Electric Commander
• http://www.electriccloud.com/products/electriccommander.php
• Jira• https://www.atlassian.com/software/jira/overview
• BlackDuck• http://www.blackducksoftware.com/products/black-duck-suite
• Gitolite• http://gitolite.com/gitolite/
![Page 30: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/30.jpg)
Strengths• Chasing the sun
• Operational teams in major timezones• Handoff procedure is well understood
• Cheap scaling• Use simple, dumb tools and relentlessly parallelize• Understand where human intervention is inevitable
• Sync, build efficiency• Custom software atop repo for very efficient repo syncs• Fastest clean builds in the business
• Automated release• Very complex branching structure is released automatically multiple
times per day
![Page 31: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/31.jpg)
Challenge: Source Code Distribution
Master
Slave Slave!!! Slave
• Single master mirroring has scaling limits• Git push is CPU intensive
• Gerrit doesn’t report well when a slave is out of sync• Custom software needed
• An independent and easily monitored sync solution is desired
![Page 32: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/32.jpg)
Challenge: Build Distribution
Build Test Approve git push
25GB x ~50/day
Boulder, San
Diego, Santa Clara, India, China,
Korea…
![Page 33: DevOps For Embedded Software Development](https://reader033.fdocuments.in/reader033/viewer/2022061114/54626d6db4af9f5d1c8b47d7/html5/thumbnails/33.jpg)
Challenge: Two-Way Integration
Internal Codeline
Integrated Codeline
Partner Codeline