Protecting the irreplaceable | f-secure.com

Scaling Software Development throughInternal Open Source Cloud InfrastructureChee-Chan KengSenior Manager, R&D

© F-Secure 2011April 9, 20232

F-Secure KL R&D and I

Keng Chee Chan

• Senior Manager2007–current

• Quality Engineering Competence

• Security Research Program

• Project Manager &Sr. Software Engineer2001–2007

• Network Management Systems

• iDEN

• CDMA

• GSM

Security Response Lab

Browser Protection

Mobile Protection

AV Engine Testing

Web Development

New Concept Development

© F-Secure 2011April 9, 20233

What did we learn from Open Source Community?

• How do most people learn about Python or Android development?

• How does someone get on board through the documentation?

• Is knowledge transfer needed when Open Source project members leave the project?

• Is staff attrition a potentially critical issue for Open Source Projects?

• Do we need mega planning and meetings to plan and design the solution?

• How many managers are needed to run open source projects?

How can companies reap the benefits of crowd in the world of open source?

Could it advantageous to companies

that have similar practice within the company?

© F-Secure 2011April 9, 20234

• Simple product lines• Less teams

F-Secure R&D’s Journey towards Internal Open Source

2005 2010

• 1 team per component• Code guarded by component teams

• Too many requests from different products• Too much planning and handover• Heavyweight design work Teams overloaded and became bottlenecks

• Product development spread across multiple countries• Major architecture changes to enable common-platform development• Proactive engineers and strong management support

• Agile Transformation• Feature Teams• More flexibility in sharing the code

• Global private cloud• Blowing it big with Continuous Integration & test automation• Internal open source with Code Guardians• Everyone has equal rights to make changes

Fro

m t

rigge

rs…

to o

utco

me

© F-Secure 2011April 9, 20235

From Component Teams to Feature Teams

Hand 1 Team Hand 2

Team

Head Team

Eyes Team Ears

Team

Leg 1 Team

Leg 2 Team

Feature Team 1

Feature Team 2

Feature Team 6

Feature Team 5

Feature Team 4

Feature Team 3

Common Practice,Common PlatformContinuous Integration

Team specific practice,Different Platforms,Integration hell

© F-Secure 2011April 9, 20236

F-Secure’s Internal Open Source Platform

• A private cloud platform to enable open source use and development within R&D

• Foundation of our product development with many people developing it 24/5

• Over 100,000 automated tests executed automatically in the cloud, either triggered automatically with code changes, or triggered daily

• Anybody can make improvements to the system, the person can be a programmer, tester, architect, Scrum Master, Product Owner, or a manager

• Examples of Open Source software used:

• Linux distributions

• Jenkins

• Eclipse

• Python + nosetests

What’s new?

April 9, 20237

F-Secure Global Private Cloud in 2010

© F-Secure Confidential

Hundreds of Virtual Machinesthat relies on Python Test Automations

© F-Secure 2011April 9, 20238

Damage Control…

Anybody can make code improvements to the system…

• But when someone checks in codes that fail the system, how do we detect and fix it?

© F-Secure 2011April 9, 20239

Stop-the-Line Radiator (Project Wide)• Information radiators indicate development have to stop for the failed components• Tests are automatically triggered whenever code changes happen• Test Automations driven by Jenkins, written mainly in Python Language• Tests should pass at all times, lots of effort and commitment required• Radiators are shown in all corridors and team rooms.

© F-Secure 2011April 9, 202310

Stop-the-Line Radiator (Team Level)

• Similar trick, but stops the failure at team level before the impact widens to other teams• Lots of effort in keeping the radiators blue

© F-Secure 2011April 9, 202311

F-Secure Internal Open Source Rules

• No single leader that dictates

• Everybody is allowed to improve the system directly

• Radiators display the build and test status all the time

• Stop-the-Line and revert commits as soon as possible

• No new features can be added without test automation

• When code breaks, it is everyone’s responsibility to get it fixed

With great power,

comes great responsibility

© F-Secure 2011April 9, 202312

How to Scale this in the Lean & Agile Way?

F-Secure’s Ways of Working:

1. Continuous Integration for fast feedback

2. Programmers and testers shares the same Test Automation system

3. Formation of effective feature teams

4. Strong management support and involvement in change management

5. Avoid dedicated central control - “It is everyone’s responsibility”

6. 2-week sprints demo to manage expectations

7. Every team owns end-to-end software development responsibility

© F-Secure 2011April 9, 202313

Internal Open Source in Action!

Contributors

F-Secure’s Internal Open Source:Scale Confidently through Sufficient Test Automation