1

Accelerated Transformation of IT

KMG International

CASE STUDY

2

Learning From Experience: Problems Created by Full Outsourcing

Cost reduction in order to increase profitability of the business became an imperative and so in 2010 the management team decided to restructure IT. Outsourcing the entire function seemed like a good proposal and so a contract was signed with an external organization that had been set up to provide all necessary IT services for KMG International. All the IT staff were transferred out and only one person remained within KMG International as a contact point for IT. The arrangements covered a flat fee plus change requests for any new projects.

However, the change requests were so numerous and complex that the overall IT costs for KMG International actually went up rather than down. Furthermore the outsourcing organization held more than 100 contracts with other technology providers in order to fulfill KMG International’s requests — adding further complexity and cost to the delivery process.

When a new management team took control of the business in 2012 they recognized that IT was a risk for the business both operationally and financially. They wanted to find a new solution for IT in order to reduce costs while improving the quality and range of services provided.

It was evident that during the period of the outsourcing agreement IT operations had run out of control. Service level agreements were lax and the contracts did not really incentivize any improvement.

Change requests coming from the business received responses late with price quotations much higher than the market. Contracts with third party suppliers were not done through a transparent tender process and didn’t necessarily reflect the best interests of KMG International.

Poor decisions were made in order to maximize the

KMG International had been growing very fast in Romania and as a result the IT infrastructure was completely redesigned between 2005 and 2009. Over that period, however, IT costs ballooned. At its peak, the IT team swelled to 140 people who covered everything from development and testing through to day-to-day operations.

outsourcing company’s short-term revenues while exposing KMG International to high risks in the long term. For example, in 2011 there was a decision to stop paying license maintenance for critical applications — in order to bank a short term cost saving. However without a valid maintenance contract in place it was impossible to escalate issues with software providers and deploy upgrades. The risk of not having an escalation path for critical defects was added to neglecting the software upgrade path for enterprise applications. In the long run, KMG International was compelled to renew the support and pay a hefty backlog of license maintenance, since critical business applications cannot function without license maintenance.

KMG International Background

KMG International N.V. (formerly known as The Rompetrol Group N.V.) is an international oil company with operations in 12 countries throughout Europe, Central Asia and North Africa. The group is active primarily in refining, marketing and trading, with additional operations in exploration and production, and other oil industry services such as drilling, EPCM, and transportation.

As of 2010, the group is entirely owned by NC KazMunayGas JSC, the national oil and gas company of the Republic of Kazakhstan. KMG International owns the largest and most modern refinery in Romania and the Black Sea region — Petromidia Navodari — which boasts a processing capacity of over five million tons per year.

The company’s retail network that operates under the “Rompetrol” brand incorporates over 1,100 fuel distribution points in Romania, Georgia, Bulgaria, Moldova, and under the Dyneff brand in France and Spain. The company employs over 7,500 people worldwide, of which around 6,000 are in its home market, Romania.

3

A New Model for IT Efficiency

KMG International management wanted to find a different solution. Having already experienced a full internal model followed by a fully outsourced model, KMG International moved to designing a hybrid model. Under this new design, certain functions mostly focused on strategy, governance and vendor management were held internally and most operational execution was outsourced.

The main objective was to reassert control for IT within KMG International so that it could direct proactively the level of service that was needed and also manage costs more effectively.

Today the Group IT function of 26 people covers the following areas:

The team acts as a business partner to the business entities that form the group. They work with various business clients to define their requirements in terms of IT solutions and services, drive the approval and procurement processes, oversee the delivery of the projects executed by with external providers and coordinate the support activities. More recently the team also took on a business development role, proposing new solutions with strong business cases to companies and corporate functions in the group.

On the operations side KMG International has a contract with SkyConsult for helpdesk, infrastructure and application support — and in turn they have an agreement with IPsoft for infrastructure support. KMG International also holds other similar agreements with various vendors to perform support tasks and provide other services.

• IT Strategy

• Systems Architecture

• Information Security

• Vendor Management

• Demand Management

• Project Management

• IT Budget Management

• Contract Management

4

Planning a Transition without Full Data Access

Late in 2012, Marcel Chiriac engaged with KMG International’s senior management and acted as a consultant to SkyConsult in order to help shape the new organization model. He took over the CIO role formally in November 2012 moving from Vancouver back to his native Romania. At that point there were three people in Group IT.

The very first step in transitioning to the new model involved exploring the 100+ contracts that had been drawn up by the outsourcer in order to discover what they were delivering. New relationships needed to be put in place, contract terms needed to be renegotiated and in many instances contracts needed to be consolidated. For example, KMG International moved from having 38 telecoms contracts to just four.

The proposed change was received with complete hostility by the outsourcing firm involved who proceeded to deny any information access to the new KMG International IT management, let alone knowledge transfer. “The entire transition to the new model was planned and managed in the same way as you might expect in a hostile takeover,” explains Marcel Chiriac.

Most importantly the team worked to re-establish control of services. The layering of sub-contracts also meant that KMG International was unable to exert any authority over many of the sub-contractors. In one example, the new CIO was refused the administration passwords for the technology that was supplied to KMG International via the outsourcing company, as they had no direct relationship in place.

His initial task was to reduce the risk in IT operations for KMG International — re-establishing an IT function that could be the business partner to the business units. Group IT would now have direct responsibility to top management and be accountable for control over all vendors. As Marcel Chirac began to plan the transition to the new model he bore three key objectives in mind:

1. Deliver a smooth transition with no negative business impact.

2. Build and implement a new hybrid model — establishing a new relationship with vendors to restore control for IT within KMG International.

3. Deliver a more financially efficient IT service.

5

The Race to Cutover

After arriving in November 2012, Marcel Chiriac had to mobilize his partners at record pace in order to take on operational responsibility for running all IT services by the contract cutover date on January 1, 2013.

IPsoft played a critical role in supporting this transition and the deployment of the IPcenter platform greatly facilitated information gathering of the IT operations that were soon going to be transferred. IPsoft brought in a very experienced team of people to work on this unique challenge. A core team came to Bucharest supported online by IPsoft operations teams in other global locations. In a very short time the team had learned and understood the new IT environment. Using IPmons they were able to find out what was present and make preparations to secure the infrastructure in time for the looming cutover date.

Despite significant obstacles to progress, such as only having a partial view of the network rather than full administration rights, a solid picture of the IT landscape was put in place. It covered the underlying details of:

• 3 datacenters

• Hundreds of servers (physical and virtual)

• 3 virtualization environments

• 4000 users

• 200 applications

• Network spanning many countries.

IPsoft carried out due diligence to manage all the underlying devices and integrated IPcenter with the other systems management tools that were in place.

3

datacenters

100s

of servers (physical and

virtual)

3

virtualization environments

4000

users

200

applications

With the clock ticking towards the cutover deadline, the CIO’s combined team of 20 people made up of his skeleton internal team, the SkyConsult team and IPsoft, prepared themselves for the transfer of IT operations supporting KMG International’s entire business.

The hostility of the incumbent provider continued to cause problems right down to the wire as many administration passwords had not been handed over. It seemed as though the team would have to shut down all servers and restart them on January 1st. With only a few days to go, however, on December 28th Marcel Chiriac managed to secure the handover of key information that would allow the service to transfer over smoothly.

IPsoft was able to find out what was present and make preparations to secure the infrastructure in time for the looming cutover date

6

Cutover Success

The strategy the team had put in place proved to be solid. On January 1, 2013 the new team moved quickly in order to secure all the systems of which it took control. This was essential and in fact there were several attempts to attack the network in the first few weeks. Happily the team was able to defend against each of these.

KMG International had only one 4-hour outage on that first day based on the failure of a single application. Given the risk in the transfer and the scale and complexity of the IT environment this was a truly remarkable feat.

Within the first month more than 2000 configuration items, such as servers and network switches, were placed under IPsoft’s management using IPcenter, including all of KMG International’s 750 plus gas stations.

Over the first six weeks the collective team deepened its knowledge of the environment in order to plan the next wave of improvements in the system. Once again IPsoft’s technology helped accelerate the information gathering. For example, a network topology diagram was created semi-automatically using IPcenter in a fraction of the time it would usually take to complete using manual methods.

Technology Scale and Scope

Vblock TechnologiesVblock is used to host virtual machines and all storage needs demanded by these systems. KMG International utilizes 3 Vblocks, one at each datacenter location.

AIX15 Physical AIX servers hosting business applications such as Oracle EBS and Web Message Broker

Linux Servers 100+ systems hosting applications such as Proxys, SAP, Maximo

Wintel Servers 150+ systems hosting applications such as Email/Time Control systems

Network Devices1000+ devices based at gas stations in Romania, Georgia and Bulgaria 400 critical devices located in datacenters and office buildings

Databases 150+ databases supporting applications such as Oracle EBS / CRM / Sharepoint

Fig. 1 — Close up view of KMG International’s ICT Environment Scope

7

0

5

10

15

20

25SLA Targeted MTTR

Achieved MTTR

Q1 2015Q4 2014Q3 2014Q2 2014Q1 2014Q4 2013Q3 2013

Applications: Mean Time to Resolve

Time Period

5.3 4.4 3.4 2.8 2.9 3.4 2.6

Achieved MTTR

SLA Targeted MTTR23.7 23.8 23.9 24 24 24 24

Stabilizing the Service

Following the successful cutover, Group IT moved into the first phase of the operational plan. This was the stabilization stage, which lasted until May 2013. During this period the team focused on the following objectives:

• Ensure that all knowledge of the environment was completed in detail.• Establish clear SLAs across all parts of the IT landscape in relation to the business.• Define and prioritize what services needed to be addressed. For example there were servers that had not been

rebooted for as long as six years, network nodes that had not been upgraded in years, databases that were never looked at. All such factors were a constant threat as they could provoke unnecessary outages so “cleaning them up” was a critical part of the stabilization plan.

The impact of the program on database performance provides an excellent example of how the service experienced by users improved. The number of problem tickets issued relating to KMG International’s two main business databases dropped by more than 50% within the first month of operations and then steadily fell further to average a more than 70% drop in tickets through the remaining months of the year.

Across all key areas Mean Time to Resolve has been cut extensively and in all instances performance has been consistently better than the new SLAs set (see tables below):

0

20

40

60

80

100

120 Achieved MTTR

SLA Targeted MTTR

Q1 2015Q4 2014Q3 2014Q2 2014Q1 2014Q4 2013Q3 2013

Infrastructure: Mean Time to Resolve

Time Period

14.2

36.1

66.8

24.818.9 13.6 12.0

Fig.2— Applications Mean Time to Resolve has fallen consistently and is well below SLA

Fig.3 — Infrastructure Mean Time to Resolve has out-performed SLAs

8

Fixing problems fast became a mantra for the team. Many good practices had been neglected over the previous years and now was the time for change. Whenever the team spotted an issue they took immediate action to complete improvements such as cleaning up logs and cache in some machines, optimizing routing on network nodes.

IPsoft moved quickly to start building runbooks and standard operating procedures that could support KMG International’s ICT infrastructure fully. This involved documenting the entire environment and defining clear change management procedures that ensured that the user experience would be protected at all times.

As a result of all the changes service levels improved significantly for the business within the first six months. Not only had far better SLAs been defined but these were never breached. The business could now rely on a solid and consistent level of service.

Whenever the team spotted an issue they took immediate action to complete improvements such as cleaning up logs and cache in some machines, optimizing routing on network nodes

8

9

PrioritySLA (Minutes) for mean time to respond

Achieved SLA Goals in 2013

Achieved SLA Goals in 2014

P1 15 7.96 5.33

P2 15 6.02 3.7

P3 15 8.04 5.79

P4 15 7.49 4.17

PrioritySLA (hour) for mean time to resolution

Achieved SLA Goals in 2013

Achieved SLA Goals in 2014

P1 4 1.9 1.18

P2 4 1.37 0.64

P3 24 1.45 1.2

P4 120 1.82 2.12

Fig. 4 — The team outperformed its 2013 and 2014 SLAs for Mean Time to Respond

SLA Performance

Fig. 5 — The team also outperformed its 2013 and 2014 SLAs for Mean Time to Resolution

The team has consistently outperformed SLAs to deliver a consistently reliable service for users. In the 16 months following the start of the programme, there have been no unplanned outages that have impacted business critical applications. Improvements have climbed as automation rates have risen.

10

Optimizing Performance

KMG International’s ambitions for IT services go much farther than stabilization. Chiriac is now overseeing a phased program of optimization in order to support the business in its future development.

The full understanding of business usage for applications and services has already allowed the team to tune the infrastructure in order to improve the service to users. For example, a program of upgrading operating systems, databases and applications was put in place, catching up with what had been neglected for many years and ensuring the group keeps up with all new upgrades as they become available.

Having consolidated telecom providers, KMG International has also ensured that network bandwidth is appropriately supplied to meet business needs at all times. Improved processes using IPcenter also mean that service is re-established much more quickly following any break in the system. When traffic is broken and a site loses connection, an IPsoft ticket immediately contacts the underlying vendor in order to resolve the issue before it becomes a major problem. Previously when a network fault occurred it would take a long time even to find out which telco provider was responsible. The automation that has been created to support this process has brought the response time for third party teams down to five minutes whereas this previously could take a couple of hours. This impacts uptime directly. Given that the number of incidents reported in a single month is around 3000, relating to ongoing monitoring of more than 750 gas stations and 30 sites, the time saving in manual labor is significant.

Most importantly IPsoft has helped move KMG International from a reactive to a proactive approach to incident management. IPmon is continuously monitoring system thresholds and identifying symptoms of potential problems that could be addressed automatically, without human intervention.

As a result of the intelligence in the system provided by IPsoft, KMG International was able to identify capacity issues with the virtualization systems and take action before running out of capacity to host new solutions. This could have been a major problem given the number of new projects that KMG International was planning to roll out over the following three to four years. IPsoft was one of the major participants in the Vblock upgrade project that allows KMG International to scale its virtual environment capacity and integrate with public clouds seamlessly, thereby eliminating for good the risk of running out of capacity.

It was a large undertaking that involved replacing two virtual environments and relocating services to a new datacenter with full disaster recovery capability. The execution began in April 2014 and was completed in September 2014. IPsoft’s team managed the application migration and optimized the capacity usage for each application on the new servers.

11

Across all applications, users have experienced the benefit of faster response times both in main sites and for the many users who are traveling and need to access their files remotely. Exchange was also upgraded and consolidated for the entire group onto a single instance that offers a much larger mailbox size. The business case for the virtualization upgrade was further improved by significant savings in software licensing. The migration involved a full review of all servers and databases, which allowed for consolidation and optimization of the licensing models.

Marcel Chiriac developed a new strategy and roadmap to 2018 in which he pushes forward with further optimization programs. SAP will be rolled out consistently across the business, implementing better processes and controls. A major business intelligence program is also being devised which will draw on the efficiencies of the underlying infrastructure. Finally, Group IT is fully enabled to take on a strategic business partner role, identifying and proposing to its business clients, technology solutions that will foster growth.

Continuous Service Improvement Through AutomationOver the past two years, KMG International has gradually increased the number of automations it runs Each quarter the number of automata that directly impact service has risen. At present, about 61% of tickets are resolved entirely without manual intervention. In addition, automation speeds up other ticket resolutions so that a staggering 91.9% of all tickets have been positively impacted by virtual engineers.

The new automata being created impact a wide range of business scenarios directly. For instance, automata have helped overcome the challenge of ensuring rapid response is provided to any of the company’s 700 gas stations no matter what time of day or night. Within five minutes of an initial alert being received, relevant issues are escalated to local third party vendor support teams. As a result, all incidents are investigated and resolved in a much shorter time frame.

In another example, automata are being used to onboard new business partners so that they can be

effective immediately. The mailboxes, email addresses and associated passwords that provide an essential channel of communication into the organization are set up within ten minutes. Previously, this depended upon a manual process and could take several hours to complete.

Significant time and effort is also being saved within the IT infrastructure team through the use of another automata that is eliminating a common cause of false alerts – mistaking legitimate downtime for systems failure. This type of problem frequently occurs when emergency maintenance activities that require services to be suspended are not appropriately identified throughout the IT environment.

In order to avoid this problem, automata implement the precise downtime notification of hosts, monitoring and other systems without any manual intervention, after receiving an authorized request from the infrastructure or application teams.

These increased automation levels translate into service quality improvement that is tangible to KMG International’s staff. In 2013, the company’s internal IT customer satisfaction index returned a score of 4.2 out of 5 from users within the business. Previously IT had been rated at less than 3.

“In just two years we have significantly reduced risk in our business by strengthening the reliability of our infrastructure and quality of our service,” summarizes Chiriac. “With the help of IPsoft we have transformed Group IT within KMG International from a reactive to proactive organization. Our business now relies on us to innovate new methods that drive efficiency to new levels. In a market that has seen the price of oil drop markedly that translates into sustainable competitive advantage.”

KMG International’s ambitious plans have borne fruit already. In 2013 Group IT saved 27% in operating expense compared with the previous year. In 2014 the focus shifted to maintaining this more efficient level of spend while improving the range of services delivered.

Efficiency Impacts Business

12© Copyright 2015 IPsoft. All rights reserved.

IPsoft Background

IPsoft automates IT and business processes for enterprises across a wide range of industries. Through its portfolio of world leading autonomic and cognitive solutions it provides services that allow its clients to secure competitive advantage. Headquartered in New York City, IPsoft has offices in 13 countries across the world and serves more than 500 of the world’s leading brands directly as well as more than half of the world’s largest IT services providers.

Email: webinquiries@ipsoft.com

US: +1 212-708-5500

UK: +44 (0)207 070 5700

Web: www.ipsoft.com

Address: 17 State Street, 14th Floor, New York NY, 10004 USA