Project-Builder.org presentation
-
Upload
bruno-cornec -
Category
Technology
-
view
180 -
download
0
description
Transcript of Project-Builder.org presentation
Bruno Cornec HP EMEA Open Source Profession Lead
Project Builder:A GPL continuous packaging solution
V2.0 April 2013
HP / OSSI / Bruno Cornec2
Introducing Myself● Software engineering and Unices since 1988
● Mostly Configuration Management Systems (CMS), Build systems, quality tools, on multiple commercial Unix systems
● Discover Open Source & Linux (OSL) & first contributions in 1993
● Full time on OSL since 1995, first as HP reseller then @HP
● Currently:● Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble
● OSL HP Advocate
● EMEA OSL HP Profession Lead
● Solutions Linux Conference and OWF board member
● MondoRescue, Dploy.org, Project-Builder.org project lead
● LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor
● FOSSBazaar and OSL Governance enthusiast
● Mandriva, Mageia, Fedora packager
HP / OSSI / Bruno Cornec3
What is the problem ?
Users / Sysadmins environment:
< 1%
< 10%
>= 90%
Users: easyness (GUI, CLI)Admins: distribution compliance, smooth integration – reduce admin load
HP / OSSI / Bruno Cornec4
Project
Flat Files
CMS/VCSCVSSVNGIT, ...
What is the problem ?
Developers environment:
autotools ?Install proc ?Soft. Eng. ?Fear of distro ?
Sourceforgeftp, web
HP / OSSI / Bruno Cornec5
Package
Flat Files
CVSSVNGIT, ...
What is the problem ?
Packagers environment:
rpmlint ?lintian ?
Distroftp, web
Does s/he only exists ?On time ? Lag behind ?Distro coverage (ver, #)Beta avail ?
CMS/VCS
HP / OSSI / Bruno Cornec6
I'm a sysadmin so I want to deal with packages only (deployment server integration)
I'm a developer so I want to ease my tests without screwing up my system and testing as a std user/admin
I'm a software engineer and want no duplication of code nor information (metadata)
I'm a a packager so I want to streamline that work for new projects I'm interested in packaging
I'm the marketing department of my own project and use packages as a “competitive advantage”
Founding no tool existing to help me doing all that (2005), so I started writing one to scratch my own itch
My answer with my various hats
HP / OSSI / Bruno Cornec7
Packaging should be a project concern as well as coding, testing, installing, .... especially for smaller projects
New mantra: “Package early, package always”
Use packaging as your only way of delivery (not a dream)Overhead minimal (not much longer than providing a tgz), maximum benefit (consistancy, reproduceability, team work improvement, distribution integration, improved deployment)
Packaging is also a marketing activity based on a technical content. It's an easy way to extend your user base, and improve your community. Makes tests easy.
THE SOLUTION IS CONTINUOUS PACKAGING
What is the solution ?
HP / OSSI / Bruno Cornec9
Project
Build +
metadata
Cloud based Continuous Packaging
VM or VE Build
Local build
RM Build Farm (may host VMs) Remote build
RepositoryServer
Local Build Server (may host VMs and/or VEs)
Packagers
Developers
HP / OSSI / Bruno Cornec10
Project
Build +
metadata
Provisioning the Continuous Packaging Cloud
VM or VE Build
Local build
RM Build Farm (may host VMs)
Remote build
RepositoryServer
Local Build Server (may host VMs and/or VEs)
HP / OSSI / Bruno Cornec11
Project-builder main goal is to help you package continuously being agnostic:
CMS agnostic: no CMS but guys it's 21st century now, SVN, CVS, Mercurial, GIT, SVK....
OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 120+ distro tuples at the moment – repositories for yum, urpmi, apt. Solaris pkg. HP-UX sd in progress
Build environment agnostic: local, VM (QEMU, KVM...), VE (rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm)
No project impact: preserves the md5sum of the delivered upstream sources. Can be completely external to the upstream project.
Goals
HP / OSSI / Bruno Cornec13
Project-builder provides additional goodies:
Easy VMs/VEs/RMs management
Macro system with perl variables to avoid duplication
Skeleton generation to help starting
Manages delivery up to your repository (ssh based)
Manages announces on mailing lists
Integration of tests in the process
Manages patches/additional sources when not upstream
Checks validity of packages built (lintian, rpmlint)
Easy creation of new versions for upstream management
Goodies
HP / OSSI / Bruno Cornec14
VM, VE & RM ManagementVE Creation:
debootstrap for .deb based ones.
rpmbootstrap (modeled after debootstrap): create chroot for multiple RPM based distributions (Mageia, Fedora, Mandriva, CentOS, OpenSuSE)
VM|RM Creation:
KVM or QEMU iso install possibilities.
LinuxCOE/OpenStack as provisioning tool for your VM|RM
VM|VE|RM setup: adds pb tools and build account
VM|VE Snapshot feature
Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg).
ssh communication for VM/RM
HP / OSSI / Bruno Cornec15
Metadata Management
Packaging process independent of the project and external
Metadata isolated - No duplication
HP / OSSI / Bruno Cornec16
./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similar to debootstrap for RPM based distributions
./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY
./rpmbootstrap/deb/control:Description: PBSUMMARY
./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY"
Macro example:
Instantiation possible from distro-ver-arch, distro-ver, distro, distro-family, distro-type, os
Similar approach for all other parameters (Cf: man pb.conf)
Configuration file usage
HP / OSSI / Bruno Cornec17
[cms|sbx]2build: Create tar files for the project under your CMS. Parameters are packages to build if not using default list
build2pkg: Create packages for your running distribution
build2ssh: Send the tar files to a SSH host
pkg2ssh: Send the packages built to a SSH host
build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and send those packages to a SSH host once built
launchvm: Launch one virtual machine
script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it
test2vm|ve|rm: Test inside a VM/VE/RM
newvm|ve: Create a new VM/VE/RM
setupvm|ve|rm: Setup a VM/VE/RM for pb usage
newver: Create a new version of the project derived from the current one
newproj: Create a new project and a template set of configuration files under pbconf
Commands
HP / OSSI / Bruno Cornec18
Project
Packaging
cms|sbx2build
build2vm|ve|rm
build2pkg
newvm|ve / setupvm|ve|rmnewver / newproj
The Big picture
build2sshDevelopers
Packagers
PackagesBuilding
pkg2ssh
HP / OSSI / Bruno Cornec19
Multi platform life cycle
User Requirements
FunctionalSpecification
DesignSpecification
Coding
Unit tests
Integration tests
Validation
HP / OSSI / Bruno Cornec21
HP+Intel Solution Brief
HP / OSSI / Bruno Cornec23
Packages
Multi platform packaging cycle
Developers
Packagers
VMs/VEs/RMs
VM/VE/RM
UUWL
HP / OSSI / Bruno Cornec24
Q & A up to now ?
HP / OSSI / Bruno Cornec25
DEMONSTRATION
HP / OSSI / Bruno Cornec26
LSB chroot support. (or not)
LXC support
Support for libvirt, virsh, openstack API
Other CMS (Bazaar, …) only when/if needed
Other VMs (VMWare, Xen, ...) only when/if needed
Multiple delivery means
Look at interactions with Buildbot
Add non-interactive signature support for .deb
Config-Model for configuration file management
REST API and Web Interface
To be done
HP / OSSI / Bruno Cornec27
Learn Project-Builder.org
Start with the Lab (63 pages)
Use man (pb, pb.conf + 8 ProjectBuilder::* man pages)
Use the mailing-list pb-announce and pb-devel at http://www.mondorescue.org/sympa
Use examples from http://trac.project-builder.org/browser/projects/
HP / OSSI / Bruno Cornec28
Project-Builder Web site / Trac / Wiki:
● http://www.project-builder.org
● http://trac.project-builder.org
Projects using project-builder.org:
● http://www.project-builder.org (of course :-)
● http://www.mondorescue.org
● Http://www.linuxcoe.org
● http://www.fossology.org
● http://kde-apps.org/content/show.php/Meganizer?content=129907
Web Resources
HP / OSSI / Bruno Cornec29
Project-Builder is mostly suited for upstream projects wanting to package their applications
Distributions provide each their build tools● SuSE: Open Build Service (Multi distro, Web based, BaaS)● Fedora (Koji)● Mandriva/Mageia (Youri, mdvsys, mgarepo...)
Other complementary tools:● Buildbot● KVM/QEMU● rpmbootstrap, rinse, mock● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, Mail::Sendmail
Project-Builder.org is one of the tool of the vcs-pkg.org initiative.
Related tools
HP / OSSI / Bruno Cornec30
”Changes are never easy to make. There is comfort and safety in tradition, but change must come, no matter how painful or expensive it may be.”
Bill Hewlett
Contact
ThanksOpen Source at HP
http://opensource.hp.com
(Linux Solution Consultant in the HP/Intel Solution Center)
http://www.hp.com/linux
Linus Torvalds, Richard Stallman, Eric Raymond, Nat Makarevitch, René
Cougnenc, Eric Dumas, Rémy Card, Bdale Garbee, Bryan Gartner, Gallig Renaud,
Joachim Langenbach among others, for their work and devotion to the Open
Source Software cause... and my family for his patience :-)