[email protected] Google, Inc. Dinah...

29
Release Engineering as More than a Part-time Pastime Dinah McNutt Google, Inc. [email protected]

Transcript of [email protected] Google, Inc. Dinah...

Page 1: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Release Engineering as More than a Part-time Pastime

Dinah McNuttGoogle, Inc.

[email protected]

Page 2: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

● Writes Makefiles

Page 3: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

● Writes Makefiles● Yells at the developers

Page 4: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

● Writes Makefiles● Yells at the developers● Maintains the source code

repository

Page 5: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

● Writes Makefiles● Yells at the developers● Maintains the source code

repository● Makes Jenkins work

Page 6: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

● Writes Makefiles● Yells at the developers● Maintains the source code

repository● Makes Jenkins work● Understands RPMs

Page 7: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

● Writes Makefiles● Yells at the developers● Maintains the source code

repository● Makes Jenkins work● Understands RPMs● Yells at the SREs

Page 8: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

What does a Release Engineer Do?

“Accelerates the path from development to operations”

Page 9: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Steps in Release Process

Check out code

Compile

Test

Release

Page 10: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

The Real Process

Check out code

Compile

Unit Tests

Deployment

Package

System Tests

Canaries

System Tests

Bug Fixes

Page 11: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

The Real, Real Process

Check out code

Compile

Unit Tests

Deployment

Package

System Tests

Canaries

System Tests

Bug Fixes

Build Artifacts

Reports

Alerts/Monitoring

Page 12: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

“Release Engineering from the Beginning”

Page 13: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Release Engineering from the Beginning

●Cheaper to put good practices in place early

●Releng, developers, testers and SRES work together

●Releng must understand the intent of how the code should be built and deployed (and then help define the processes)

●No “Build and Pray”

Page 14: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

Page 15: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

Page 16: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

● Automated Build System

Page 17: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

● Automated Build System

● Build Identification Mechanism

○ Unique build strings

○ Ability to identify when/where/how a binary was built

Page 18: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

● Automated Build System

● Build Identification Mechanism

● Packaging

Page 19: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

● Automated Build System

● Build Identification Mechanism

● Packaging

● Deployment

Page 20: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

● Automated Build System

● Build Identification Mechanism

● Packaging

● Deployment

● Reporting/Auditing

Page 21: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Releng Building Blocks

● Source Code Management

● Build Configuration Files

● Automated Build System

● Build Identification Mechanism

● Packaging

● Deployment

● Reporting/Auditing

● Best Practices

Page 22: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

End Results

●Continuous delivery of new products

●Early bug identification

●Repeatability

●Enforcement of policy and procedures

●Hermetic build process

Page 23: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Hermetic (Dictionary.com)

1. made airtight by fusion or sealing.

2. not affected by outward influence or power; isolated.

3. of, pertaining to, or characteristic of occult science, especially alchemy.

Page 24: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

So what does a release engineer

do????

Page 25: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Disciplines within Releng

●Automation and Execution

●Consultation and support

●Source code repository management

●Tools development

●Audit compliance

●Metrics

●Best Practices

●Deployment

Page 26: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Skills and tools - then

2004● make, ant● cvs, svn● cruisecontrol, home

grown scripts● Machine-oriented

build/deployment● Scripting● Sysadmin skills

Page 27: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

Skills and tools - then and now

2004● make, ant● cvs, svn● cruisecontrol, home

grown scripts● Machine-oriented

build/deployment● Scripting● Sysadmin skills

2014● maven● perforce and git● Jenkins, Chef,

Puppet, bamboo, buildbot, teamcity

● Container technology● Object-oriented

programming● Sysadmin skills

Page 28: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

The Future of Releng

● Industry-standards for:○ job ladders and job descriptions○ best practices○ metrics○ compliance

● College curriculums

● More solutions from vendors○ end-to-end

Page 29: mcnutt@google.com Google, Inc. Dinah McNuttflowcon.org/dl/flowcon-sanfran-2014/slides/DinahMcNutt... · 2014. 9. 3. · Reporting/Auditing Best Practices. End Results Continuous delivery

And of course - conferences!

● Usenix Release Engineering Summit (URES) 2014○ East - June 20, Philadelphia○ West - November 10, Seattle

● IEEE RELENG 2014○ April - Mountain View, CA

● Flowcon 2014

○ September 3-4, San Francisco