DevOps

Finding the right amount

#NercompPDO3

Who am I?Ian Rifkinirifkin@brandeis.eduDirector of Data and Systems IntegrationBrandeis University

- Web and Middleware Development- Identity and Access Management- Data Warehouse- Data Integrations

About BrandeisFounded in 1948

Waltham, MA (9 miles west of Boston)

Undergraduate enrollment: 3,635

Graduate enrollment: 2,087

Staff FTE: 1,150

Faculty FTE: 569

Finding the right amount?What do you already do? Even consider informal practices.

Where you want to see your team and why?

Consider: How much is appropriate for your environment?

We do not have any DevOps software or practices.

My teamDevelopers that are responsible for deployments as well as responsible for the overall deployment environments.

Dev + Ops = just trying to get everything done

Is this a problem?

What goals really matter to you?Maintainable

Repeatable

Minimize downtime

Limiting production support, especially off-hours

Cross-training / coverage

Let’s take a closer look at DevOps concepts applied to my current practices…

Plan - status meetings, tickets, project managers as needed

Create - development standards

Verify - quality of our testing/verification varies on project and stakeholders

Package - Click a button, get it done

Release - Click a button, get it done (also: standards, standards, standards!)

Configure - Automatic when possible; responsibilities cross group borders

Monitor - logs, splunk, uptime robot, hyperic, alerts to email and Slack

Applying the DevOps Toolchain to My Reality

OK, maybe we were doing DevOps before DevOps was cool. #OldSchoolCool #DevOps

PlanningGathering requirements early and often

Change management (mitigating scope creep)

Status meetings! Consider format; hold people accountable; while making it a meeting people look forward to attending

Let’s talk about standardsDirectory structure (where to find our software)

Software builds

Code structure

Software repository

Release management mechanism

Directory StructureOnce you know how we organize our software you can navigate to any of it with ease.

/usr/public contains underlying software (Apache HTTPD, .so files, OpenSSL, etc.)

/var/applications contains the applications

/var/data contains database files, logs and other data files

Software runs as “distid” user (not root)

The same setup is true on any server within our purview.

Underlying Software Builds (/usr/public)Download third-party kits to “build server”

Giant “make” file

Process creates /usr/public

Defined process to sync to target servers

Super easy, repeatable process to add software or do one upgrade

Extremely painful process to upgrade everything

Code Structureetc directory for config files

.control directory for encrypted passwords

bin directory for scripts (e.g. cron jobs)

templates directory for HTML templates

website directory for CSS, JS, images

README

and…

Perl Code Structure - Basics/var/applications/[App_Name][ver]/perllib/

Brandeis/

[App_Name]/

Handler.pm

View.pm

Process.pm

Schema.pm

Brandeis::App_Name::Process

Python/Flask Code Structure -- look familiar?In addition to same standards (e.g. bin, etc, templates, README)…

config.py Config_sandbox.pyForms.py __init__.pymodels.pyprocess.pyviews.py

It’s all about standards.

Software RepositorySubversion (svn), Locally hosted - Future: Perhaps github

Gets the job done, but conceptually getting outdated and no reason to be on-site.

Source control doesn’t replace the need for communication between team members.

VerifyQuality of our testing/verification varies on project and stakeholders

Challenges of developers being testers

Challenges of stakeholders being testers

Hired one QA staff in Project Management team for everything

Concept of Quality Champion

PackagesCustom Web-based UI to build deploy software packages

Click a button to build a kit/package

Click a button to release any version

Click a button to restart a process if needed

Desire to replace with something not custom but not complex.

Click to deploy

Software Releases/var/applications/[production-name]/[VER#]

/var/applications/[production-name]/production → /var/applications/[production-name]/[VER#]

Not required to keep every old version on target server but can be convenient.

Automatic creation of environment specific symbolic links (e.g. /var/applications/[production-name]/production/etc/example.xml → /var/applications/[production-name]/production/etc/example.xml.DEV)

Know what’s deployed

Locally Hosted Third-Party SoftwareRelease to /var/applications (link from other location if needed)

Separate configuration from application -- Goal is to be able to upgrade software and overlay configuration.

Create a repeatable process (even just a simple script to create symbolic links)

ConfigureUnlike software releases, this crosses into our server provisioning group

Important to have close relationship across groups

Server images (minimal vs. one with some of our basic setup and SSH keys)

Especially if minimal, manual script to walkthrough to get to our “standard”

Could be better, but at least it’s documented

MonitorLogs

Splunk

uptime robot

Hyperic

alerts to email and Slack

Building the culture of collaboration.

Everything is changingPreference tobuy vs. build

Preference for cloud/SaaS solutions

PeopleSoft → Workday

Increasing need forIntegrations (iPaaS tool)

Applying DevOps in low-code areas?Data integration software with a graphical user interface

Easy to click, drag and deploy…

But how to maintain quality?

How does this fit into DevOps practices?

Consider product limitations (e.g. what if you cannot create builds?)

Room for improvementBuild server and make file vs. package management

Custom release management UI vs. cloud products

Local Subversion vs. GitHub

DevOps for iPaaS

QA / testing (let alone continuous integration)

DevOps for SaaS (e.g. Workday)

Personal

Team

Other groups and outside individuals

Take time to appreciate successes

Keeping the conversation goingHow can we help each other?

Ian Rifkinirifkin@brandeis.eduDirector of Data and Systems IntegrationBrandeis University

Please complete your evaluation emailed to you!

Thank you!