#

Joshua BloomSr. Build & Release Engineer

Managing Commit and Release Decisions with “JEST”

#

• What are commit/release decisions?• How do we determine readiness?• What challenges do we face?• What solution can we adopt?• Why do it that way, and how can we improve it?• Q & A

Topics

#

Commit and Release Decisions During Software Development

“You must be the change you want to see in the world.”

— Mahatma Gandhi

#

• Authorizes the promotion of a software entity to a higher state of readiness/completion.– “Commit” reflects endorsement by the approver, and the

versioning change in the depot.– Can occur at various points in the software development

life-cycle.

• Release decision is the special case that authorizes distribution of software to end-users.

Commit Decision

#

Examples

Copying up from a development branch/stream to an integration

branch/stream.

Copying up from an integration branch/stream to a release

branch/stream.

Submitting into any controlled branch/stream.

Signing off a document.

Updating a web site.

Delivering a software package to end-users.

Commit Decisions Release Decisions

#

Determination of Readiness

“It is in your moments of decision that your destiny is shaped.”

— Tony Robbins

#

• Some commit decisions may require more than just a code review (e.g. release decisions).

• Testing alone cannot determine readiness.– Coverage may be incomplete.– Failures may be incorrect: faulty test cases or bad data.– Software isn’t perfect; there are always problems to be

resolved later – or maybe not at all.

• Different stakeholders have their own interests.

Completion/Exit Criteria

#

Who in your organization helps decide when software is ready for promotion?

Decision-Making Stakeholders

Engineering• Dev • QA

Upstream • Marketing • Business Dev

Downstream• Sales• IT• Tech Support

Regulatory• Finance• Legal

#

• All stake-holding teams contribute to the decision.– All teams vote explicitly; no abstentions.

• Who represents each team may vary, depending on the decision.

• Usually, the decision must be unanimous.– Approval may be conditional upon future changes.

Rules for Stakeholder Decisions

#

• More release events occur per unit of time.• Release events happen independently of

development cycles (e.g. Agile sprints).• Development efforts split into parallel tasks (often

in separate development-level branches/streams), to avoid release dependencies.

Releasing with Continuous Delivery

#

• Integration into release-level branches/streams occurs as late as possible.

• Each task is released when complete, within a planned delivery window. Specific delivery dates are no longer predictable.

• Release packages are smaller, with more fluid content.

Releasing with Continuous Delivery

#

Periodic vs. Continuous DeliveryPeriodic Release Cycle, of Major and Minor Releases:

Asynchronous Release Schedule, with Continuous Delivery:

N.0 N.0.1 N.0.2

N.1 N.1.1

#

The Challenge

“We’re gonna need a bigger boat.”— Roy Scheider in Jaws

#

Case History: Releases at 23andMe

Company founded

Adoption of monthly release schedule + ad hoc updates.

Controlled branches (integration and release) are managed by manual email requests to “toggle” protections on and off again

First product release

2006

2007

Adoption of weekly release schedule (alternating major / minor) plus ad hoc updates.

20092008

2011

#

Case History: Releases at 23andMe

Development team adds Git Fusion, for their members who prefer to use Git.– Re-pull generates new sha1sum– Re-push generates new change number

Semi-automated management of controlled branches with counters, email, and submit triggers

2012

• Outside audits of release process

• Development of JEST

2013

2014

#

• Stakeholders may be asked to authorize a release at any time.

• Stakeholders may be asked to authorize more than one release at a time.

• Stakeholders may need some time to make each decision.

Asynchronous Release Decisions

#

For historical, legal, and other reasons, a record of the release decision should be kept:

Record-keeping Requirements

• Exactly who authorized the release?• When did the authorizations occur?• Were there any special

considerations?

#

• To manage changes in a controlled branch/stream, the submitter must have write permission, but:– Stakeholders must still determine readiness, and grant

approval.– Records of the decision must be kept.– Toggling permission on and off is tedious, and doesn’t

provide granular control over individual change sets.

• So, permissions alone are not sufficient.

Limitations of Permissions

#

The Response

“In all chaos there is a cosmos.”— Carl Jung

#

• “Version everything.” – Perforce Motto– Ensure that every commit/release decision involves

submitting a change in the depot.• May require a placeholder file (e.g. a “release spec”).

– Also, record the details and history of each decision in the depot (rather than in a separate tool).

• Establish triggers to:– Control the flow– Keep everyone informed

Solution Overview

#

• Partition: A tree of files (often a branch/stream) in the depot, for which submissions will be controlled

• Owners: Perforce group(s) that identify the members of each stakeholder team for the partition– List can be static, or determined at submit time

• Approver: Specific user in the Owners group who granted/declined the approval request

Key Concepts

#

• Partition policy: A set of choices for processing approval requests within each partition:– Restricted hosts (optional): Only allow approvals from

client hosts with certain IP addresses.– Self-approving (yes/no): Bypass creation of the approval

request, when the Submitter is one of the Owners.– Duration (optional): Default amount of time until an

approval expires. Can be overridden for each request.– Deletion (optional): Auto-delete request after submitting.

Key Concepts (cont’d)

#

• Perforce job records provide a means for storing structured meta-data (fields and values).

• Designed for tracking defects (optional), so isn’t used in many organizations.

• An extensible schema (“p4 jobspec”) allows jobs to be easily repurposed for other uses.

Jobs: the “J” in JEST

#

Field Type Purpose

Job word Unique name for the approval request.

Status select: new | pending | granted | declined | submitted | expired | closed

State of the approval request.

Submitter

word (user name) The user who is submitting the changelist to be approved.

Change word (change number)

The number of the changelist to be approved.

Approver word (user name) The user who granted or declined the request.

Owners word (group name) A group of users who approve changes in the partition.

Partition word A named subtree of files in the depot.

New jobspec for Approval Requests

#

Field Type Purpose

Limit word (number) The number (count) of changelists covered by the approval.

Expiration date Time when the approval expires (optional).

Comments text User-supplied comment about the approval request.

Updated date Auto-maintained date of last modification to the request.

Checksum word (sha1sum) Supplied by Git Fusion in its changelist descriptions; helps match the approval to renumbered changelists.

Files bulk Approver-supplied constraints on which files in the partition can be submitted (optional).

History bulk Auto-maintained log of all updates to the approval request.

New jobspec for Approval Requests

#

• Sent automatically by a trigger, whenever the approval request (job) is updated.

• Goes to the Submitter, and all Approvers named in the Owners group for the Partition.

• Text includes:– The approval request, as specified by the job record.– The corresponding changelist.– Instructions for granting/declining approval.

Email: the “E” in JEST

#

One omnibus script used for multiple triggers:• change-submit: Blocks the submission until it has

been granted approval by all stakeholder teams.• change-commit: Finalizes the changelist details.• form-in (job): Validates all updates to the request;

sends emails.• form-delete (job): Validates user deleting the job.• daemon: Sends reminders for pending requests.

Submit Triggers: the “ST” in JEST

#

1. A user attempts to submit a changelist in a pre-defined Partition of the depot.

Process Flow

#

<

2. The change-submit trigger fires, and:a. Identifies which group(s) of stakeholders (Owners)

need to grant approval in the corresponding Partition.b. Cannot find any approved request(s), so it creates a

new one for each Owners group in the Partition.

Process Flow

#

ƒʹ<

3. For each new job, the form-in trigger fires:a. Validates the job fields, updating them as necessary to

match the changelist.b. Emails instructions to the members of the Owners

group (and to the Submitter), how to grant approval.

Process Flow

#

4. After the job record(s) have been created, the change-submit trigger unlocks the files in the changelist, and exits with a non-zero value, effectively blocking the submission.

Process Flow

<

#

5. Sometime later…For each pending request, an authorized Approver (any member of the Owners group) updates the job record, granting approval.

Process Flow

#

6. The Approver’s updates to the job record fires the form-in trigger, which:

a. Re-validates the job fields.b. Sends notification of the approval to the Submitter

(with a copy to all Owners).

Process Flow

ƒʹ

#

7. After all requested approvals have been granted, the Submitter resubmits the changelist.

Process Flow

#

8. Once again, the change-submit trigger fires, and repeats its search for the necessary approvals. The trigger finds the granted approvals, and it allows the changelist to proceed.

Process Flow

<

#

9. Upon successful submission of the changelist, the change-commit trigger fires, finalizing fields in the approval request:

a. Change field is set to the renumbered changelist.b. Limit field is decremented.

Process Flow

>

#

10.Again, the update to the job record fires the form-in trigger, which:

a. Re-validates the updated fields.b. Notifies everyone that the changelist is complete.

Process Flow

> ƒʹ

#

11.The change-commit trigger resumes. If the approval allowed multiple changelists, then a new job record is cloned, whose Status is already granted, and with a decremented Limit.

Process Flow

>

#

12.Creation of the new request fires the form-in trigger again, which:

a. Validates the fields in the cloned job.b. Sends notification of the approval to the Submitter

(with a copy to all Owners).

Process Flow

ƒʹ>

#

13.The change-commit trigger resumes. For some Partitions, the approval requests are deleted.

Process Flow

>

#

14.Each attempted deletion of a job record causes the form-delete trigger to fire, which ensures that the Submitter is allowed to delete that job record.

Deletion occurs.

Process Flow

> ƒˉ

#

15.Control returns to the change-commit trigger, which exits successfully.

Process Flow

>

#

• Pre-approval:– Submitter or Approver can create a request in advance.– The change-submit trigger connects the request to the

change.

• Multiple approvals:– Approver can grant approval for multiple changes.

• Approval expiration:– Approver can set an expiration for the approval.

Variations

#

A daemon runs periodically, searching for job records whose Status is pending. If any are found, reminder notices are sent to the members of the corresponding Owners groups.

Daemon activities

#

1. Identify which depot Partitions will be controlled.2. For each Partition, identify its Owners and other

policy details.3. Update the jobspec spec.4. Install Python interpreter for the trigger script on the

Perforce server, if necessary.5. Set locale-dependent variables in the trigger script.6. Update the Triggers table.

Deployment into Service

#

Name Type Path Command

# Separate pair of submit triggers for each Partition path in the depot:

unique change-submit //depot/path/... script-path pre-submit %user% %change%

unique change-commit //depot/path/... script-path post-submit %user% %change%

…

# Two form triggers total, for jobs:

unique form-in job script-path save-job %user% %clientip% %formname% %formfile%

unique form-delete job script-path delete-job %user% %clientip% %formname%

Trigger Table Entries

#

Design Choices and Enhancements

“Wisdom consists of the anticipation of consequences.”

— Norman Cousins

#

Why not use Swarm (or some other Perforce entity, such as counters or text files) instead of jobs?• Jobs have built-in, extensible fields.

– Easy to parse and validate the relevant information.

• Swarm’s reviews focus on file-level coding issues.– JEST manages higher-level readiness decisions.– Can use Swarm with JEST: Require code reviews as a

condition for approval, among engineering stakeholders.

Rationale for jobs

#

Should JEST trigger on shelve events, too?• Submit events already cover all situations.• Some situations will not work with shelving:

– In cases of self-approval, user should submit because there will be no block.

– Git Fusion cannot shelve, so git users do not have that option.

Triggers: Submit vs. Shelve Events

#

• User submits change to a gateway branch/stream.– Compare to Swarm’s shelving approach.

• Change-commit trigger sends approval requests.• Daemon detects approval, and …

– Merges down any new changes from the controlled branch/stream to the gateway branch/stream.

– Copies up the new change from the gateway branch/stream to the controlled branch/stream.

– Closes out the approval request.

Enhancement: Single-Phase Submit

#

• Build a custom set of web pages for Submitters and Approvers to find requests, check/update Status, etc.

• Supports electronic document signatures that require immediate authentication, via:– Web site option to force user login at time of approval– Partition option to allow approvals only from web site

Enhancement: Web-based Access

#

Questions / Feedback

“By questioning we arrive at the truth.”

— Peter Abelard

##

Thank you!Joshua Bloom

jbloom@23andme.com

#

Joshua BloomSr. Build and Release Engineer23andMeJoshua Bloom has worked at a number of successful Silicon Valley software startups that have used Perforce, including PlaceWare (web conferencing), Trigo (product information management), and 23andMe (personal genomics). As a build and release engineer and Perforce administrator, he keeps the depots running, educates novice users with tips and tricks, and implements automated solutions with third-party tools. And, yeah, when time permits, he runs the product builds and releases, too!

(Speaker photo here)