Andriy Samilyak

Kaizen Magento Support

Dzien dobry!

Dzikuj bardzo ze mam mozlywosc movic dla was dzisiaj

Today I would like to tell you few stories about the thing that you may not like doing at all supporting clients and projects.

numbers

50+ projects 10+ Magento partners 45.000+ issues solved

In OpsWay we are working with many merchants and as a support backend for Magento partners around the world.We spent quite a time doing this for our clients and it seems to me that we have some ideas worth sharing.

Kaizen == change for better

So, Kaizen Magento Support First of all, what is Kaizen? This is just Japanese word for improvement and it is used to described a set of practices introduced to manufacturing in Japan after II World War.It is considered that Japan had success with this approach after all!

The core of Kaizen is the idea that things are not ideal, never, but the key is to change things constantly in order to make them better.You do not have always a big plan, but you know the best next step.

This is an approach we propose to all our clients: launch early and then see what you need to introduce it quickly.So Kaizen is about being like water constantly changing to better suit realily.

Kaizen

Small steps

Gradual

Anticipate mistakes

Do not know final destination

It is all about feelings

Urgent: SortingURGENT Seal for SSL certificateDY content UrgentUrgent: Email SubscriptionUser is getting confused urgentUrgissimo: Homepage BannerStrange and urgent bug!VERY URGENT: Travel WrapsUrgent 17% VAT

We found that most of our clients do not want to hear this at all. Priorities are shifting rapidly and we become partly an obstacle for our clients with such approach. Just take a look at these titles. BTW, these are the real subjects of issues created by our clients.

It looks like our clients, the same as us, take decisions more guided by feelings (or you can say we are irrational)

So at the very beginning of our services we decided to go the way client choose: be it wrong or right.It is tough, as we have to ensure stability in the environment of complete change.

Along the years we found that some things (we call it practices) works for this goal, some do not.

This presentation is about what works for us take it as it is, sweat and sleepless nights are in every word here.

Testing

So, lets keep in mind that clients tell what to do. What WE can do it?The usual answer in software companies is planning : planned deliveries, roadmaps, batches, more and more testing.

We tried Testing in support as well: external QA, QA team members, we even set up a new QA company.But we found that even with greatest effort we can not ensure that something works 100%.

Here is a story.Of one of Gold Magento Partners where we are involved for support, has a very strict deployment pipeline, which by the way includes testing.This pipeline works fine for development, but when it comes for support (read it fast) changes .

Testing myth

SupportDevelopmentFixed a bugTested everythingUpdate on Menu moduleDeployment10minsAAA!URGENT!PROBLEM!This Magento partners has projects compiled totally from independent modules. So one day, we fixed a bug and tested that everything works fine.Just at the same time, another team, in another part of the world, working on another project made an update for module used in supported projectHaving no idea about this we deployed a fix and broke everything.

You may thing it wont happen to you. And you are right exactly this, wont happen to you, but there are myriads of other problems that may and that will happen,This brings us to very important conclusion:

Shit happens

Im sure you know this guy

The takeout for us it that we can not think of all that comes, It happens! so you need rules to work WHEN it comes.

There was something more important than being bug free

Communication matters

Rule #1 for us is Communication matters. You are judged not by results but by how you present these results!

Here is another story:Rogel and StageStore admin story

3:01am EST Lost access to project

3:05am EST Client informed

3:10am EST Client informed about a progress

3:34am EST Hosting support replied (client informed!)

3:45am EST Problem is solved! (of course, client informed)

9am EST Guys, thank you for your support!

It was a bright summer morning, when due to a human error we lost completely server access to one of the US-based projects just before a big sale.Once again, it was completely clear that we were a reason for the problem, It happens.

We informed client immediately using all means (email, SMS, skype messages) and told what we were doing.Evidently, no one was reading our mails, as it was a deep night in New York. We were looking for a what we can do furiously constantly updating client.In half an hour we finally had a much-needed help from hosting and problem was solved quickly.

Later, when C-level executives read our mails having a coffee in the morning we had this: Guys, thank you for your support

That broke my world completely: it was our mistake but someone said Thank you! Why?

There were two takeaways for us: communication is the king and most of all it should be timely, meaning fast reply and reaction.

Communication matters

Everyone can reply to client

in English

comprehensible

with no apparent mistakes

We built-in utmost importance of communication into our HR cycle:

writing test in EnglishEvery engineer has to write a short essay (really short one) on technical subject.

BTW, it looks like those who can put ideas in words on average are better developers (ok, I got no prooflinks, just a feeling)

KISS

KISS

&

Find a problem before your client

Test crucial use cases manually

Automate testing

Alert team on errors

... on any error you define!

We built-in utmost importance of communication into our HR cycle:

writing test in EnglishEvery engineer has to write a short essay (really short one) on technical subject.

BTW, it looks like those who can put ideas in words on average are better developers (ok, I got no prooflinks, just a feeling)

Check what is important

... for your client

%

$

DevOps

Culture, culture and, finally, culture

Everyone is equal

Everyone has access to everything

If you can do something for client do it now!

Boring tasks are for computers

But just telling Accepted, will fix it and get back to you ASAP does not help as you know.So the next rule to come is Everyone should be DevOpsThere is no Code words, but we need sysadmin to fix something.

Ok, get me right we have engineers who are better skilled in system administration, but they also reply to clients and are involved in development issues.

From the first test on interview, we judge about engineers both as developers and operations people.

No passwords

MagentoAWS all the rest (~20 applications)JenkinsSSH

No wiki for credentials

We went on evolving our devops infrastructure and based it on opensource or made it opensource.

As much as possible we threw away documentation, at least part of credentials and access. You know this server login/password, etc

We treat documentation as an application that needs maintenance itself, so less applications less hussle.

Chef knows everything!

With just one command you get immediate access to servers, even if you do not know IP and user.

Onelogin

I highly recommend using Onelogin as Identity Provider. Most of our passwords do not exist and can not be stolen. You do not need to change password when you have staff changes.

Moreover, using Onelogin we can faster add new engineers: you just add access in Onelogin and the rest like access to our issue tracker, time management application, wiki and so on is done automatically.

Magento Onelogin

Identity provider like Onelogin, enables us to securely manage access to all our client Magento backends.

We install a module, that engineers are using to login to backend without knowing store-specific passwords.

Magento Onelogin

You are using only one login/pass to access all our projects.In case of necessity, this login can be disabled in minutes.

Moreover, Onelogin keeps history of logins with can be used for audit purposes.

Magento Onelogin

Feel free to use this module.

Sandbox in minutes

You can imagine that working on dozen of projects simultaneously means that you need to switch fast.As soon as we do not have PC or Mac given to new engineers when he or she comes to OpsWay, we can not even ensure that it will be fast enough to run projects.

In fact I know that some of our engineers works on ChromeBook and I wish everyone work as effectively as these guys do.

One of the tricks is that everyone can run personalized virtual server with complete copy of project environment on powerful hardware.

We create new images from test environment often. And with Jenkins everyone can set up new server within 2 or 3 minutes.

Then you can ssh to new server and remotely synchronize code or assets to it.

One button deployment

Of course, we use Jenkins in more conventional ways, like every project can be deployed with just one button.

Simple sandboxing

Of course, we use Jenkins in more conventional ways, like every project can be deployed with just one button.

24x7

Enough about tools, lets see what is next about rules we came with.

Having good people around is not enough, as even the best get tired and need to sleep.While it looks like our clients do not sleep at all, making money.And I told you that starting communication right now is essential for good customer feelings.

So quite soon, in the very beginning of our support services we understood that we need 24x7 service in order to be available when clients need.

BTW, you do not always need super-duper expert at night. As always, communication first and devops next.

The benefit is obvious: we do not need to wait for special delivery days or to be afraid of Friday deployments.If it is necessary for clients changes are coming when they need it.

Follow the Sun

But to say 24x7 is easier then to do it.We tried different approaches

24 hours shift

=> 2 developers lost

The same developer over night during a week

=> one client + 3 developers lost

c) Only one thing is left as effective Follow the Sun.But what to do if you are not worldwide enterprise with offices around the world?

The answer is obvious be like it!

Remote team

We started with Far East of Russia, it was Vladivostok first, then came Irkutsk. No we have more then 15 cities we have engineers from OpsWay.

Nowadays we here more and more about remote teams. Here is what IS remote for us: two dozen peoples working across different timezones and thousands kilometers.

I think this covers the rules we found that are working, but yet I have few words to tell about thing that do not work

Thank you!

Andriy Samilyak

ansam@opsway.com
skype: samilyaka