Bring Your Project From a 10 Years to a 3 Months Release Cycle

Stefan Schmidt

Agenda

Introduction

Release History

Technological Side

Social Influences

Next Steps

Summary

Introduction

Enlightenment is not a web project

Instead it is written in plain C

Enlightenment is not developed by a StartUp

Instead it is developed by the FOSS community and companies such as Samsung

Enlightenment is not developed by a 5 person team in one office

Instead it is developed by an international team distributed over many timezones

All this might result in different understandings of agile development :-)

Release History

Release History

The Enlightenment project started in 1997 by Carsten Haitzler

Some early releases until E 0.16

A 12 years gap from first code to release E 0.17

Building a whole new set of libraries for it but no releases for them either for a long time

More release driven now but still feature based and not time based schedule

Reasoning

Most of the active developers at that time did not see a need for releases

It was a project driven by volunteers which used the code straight out of CVS/SVN

Some argued for it as the fastest agile development cycle ;-)

Nobody stood up for taking care of the releases so it did not get done

What Changed?

Different people joined the project which had more interested in releases

Most Linux distributions wanted us to do releases before including our code

Commercial parties also asked for releases

With the pressure also the understanding grow that releases would help the project forward

How did we do it?

That is what the rest of the talk is about

From the technological side we improved our tools and infrastructure

On the social side we steadily worked on giving releases a more prominent role

Technological Side

Tools and Automation

One of the biggest roadblocks is to build trust in the code base you are going to release

In the Enlightenment project with the two main libraries alone having around 850.000 lines of code

You need a variation of tools that will help you to build up this trust

SCM, automated builds and maybe continuous integration, static analyzers just to name a few

SCM and Understandable Commits

As an very old project CVS was used in the beginning

Switch to SVN for a newer tool but same old practices

Large amounts of work committed as one change

Many unrelated changes at once

Switch to git

After some struggling it helped to allow people working on different things without dumping everything into master

Improved overall commits, code as well as commit messages

People start to develop larger changes in branches

Automated Builds

Setup automated builds with Jenkins

Quick builds for every change pushed pushed to master

Building for x86, x86_64, x32 with gcc as well as clang and also cross-compilation for windows with mingw

Very useful as many developer only use one system for their work

Many of the discovered problems are getting fixed instantly (see social pressure)

In the dark times before we had this setup builds have been broken for some configurations for months

During the nightly builds we run time intensive things like static

analyzers and other testing

Static Analyzer

Static analyzer can be very helpful for a plain C project

Started with scan-build, part of clang, from the LLVM project

Output quite noise and sadly a lot false positives

Also experimented with klocwork but the proprietary nature made it hard to share results with the community

Now also using Coverity which is also proprietary but offers a free service for FOSS projects

Especially Coverity uncovered quite some issues which have been overlooked by code review

You still have to deal with false positives but the signal to noise ratio is way better

Test Suite

We have unit testing for a lot of components

But with 358 tests we only cover 29.4% of our lines of code

We are an UI toolkit so work is also ongoing on pixel perfect regression testing

We are still weak on the automated testing side and need to improve

One of the bigger worries before every release

Build Complexity

By history the Enlightenment project was happy to add configuration options

Over the last two years we tried to reduce these options to something maintainable

Merged libs allow easier maintenance. 11 different libs have been merged into EFL

All of these actions had direct influence on how much different configurations we can test

The reduced set of libraries also eased the release process

Social Influences

General Social Changes

Once we had IRC and email notification about broken builds social pressure grow to fix them

Keep people in the loop and highlight positive changes (e.g. weekly QA newsletter)

Most people have been ok with adapting to changes but you have to provide docs and guidance

People follow the lead but would not have done it on their own. (e.g. backports of fixes to stable branches)

Towards Releases

Easier once we came over a long time without a release

Getting into the habit to do releases more often

Core application now do releases a while after the libraries

Happened naturally

Switch to time based releases. Now fine tuning the process to identify problems

Took over as release manager (Only because I build up trust to the code base due to all the automation)

There need to be proof that it will really work

Invest time to make a sane and realistic schedule

Next Steps

Changes to come

ABI checker to avoid breakage surprises after a release

Jenkins job that builds a magic developer branch

Allowing developers to tests their changes for many configurations

Thinking about adding arm builds to jenkins

Thinking about running automated valgrind runs to detect memory issues (-fsanitize* could be an alternative)

Fine tuning of the release process

Think about regression test. Having them mandatory for fixes might blow our man power.

Integrate the pixel perfect regression tester with jenkins for fast feedback

Changes that might never come

Every commit runs through our jenkins testing before it goes into master (Qt project does this)

More code review for core developers before changes hit the repo

Hard to bring in as many developers want to avoid any delay before the code hits the repo

Automate the release process (NEWS generation, announcement templates, upload tarballs, etc)

A simple version bump and pushed git tag might be enough to get it all done :-)

Summary – What you should take home

Automate as much as possible to keep the workload down and the project always in a good state

Invest in all kind of tools to make your job easier

Show the benefits of more regular releases and proof that it is possible

Keep on fine tuning the process to avoid as much problems as possible

Expect that people will only follow if you give them the lead and show how the process could work

Thank you.

References

http://www.enlightenment.org

http://build.enlightenment.org

http://git.enlightenment.org

Imagery sources

http://www.linuxtag.org/2014/

http://www.enlightenment.org/

https://openclipart.org/detail/192629/gear-tools-by-ben-192629

http://git-scm.com/downloads/logos

http://jenkins-ci.org/

http://www.coverity.com/

https://openclipart.org/detail/26498/present-blue-pack-by-minduka