Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.

PUBLIC INFORMATION

Mallinckrodt: Pharmaceutical Plant Saves

$100k with Virtualized PlantPAx® Platform

Don Geers

Senior Maintenance Engineer

Mallinckrodt Pharmaceuticals

Agenda

Company Overview

Project Team

Our Challenge (2 phases)

Justification

What’s Out, What’s Left, What’s New

Virtualization

Factory Acceptance Testing

Qualification & Startup

What Went Well

Lessons Learned

Going Forward

Q&A

2

Company Overview

3

Pharmaceutical manufacturer

Bulk Active Pharmaceutical Ingredients (API)

Specialty Chemicals

Imaging agents

Drug chemicals

Based in St.Louis Mo

140+ year old facility on 40 acres

500+ Employees at the St.Louis plant

Other Manufacturing Sites

Hobart NY, Raleigh NC, Dublin Ireland,

others

FDA Validated Facility

Long term data retention is critical

Project Team

Mallinckrodt Pharmaceuticals Don Geers – Lead Controls Engineer

Lead Process Engineer

Various other Process Engineers, Validation and Quality

personnel.

Rockwell Automation

Jim Wardell – Lead Project Engineer

Minimum of 6 additional engineers for support

Local Rockwell Automation engineering support – Long term

4

Our Challenge

Phase 1

Develop a programming standard for all new PAC based

installations in our facility. Provides for a common

programming structure across our different manufacturing

units.

Phase 2 (the upgrade)

Replace a legacy HMI and Allen-Bradley PLC3 / PLC5 based

Control System.

5

Justification

How did we justify this project.

Cost avoidance.

Process runs 24x7x364. Uptime is extremely important.

Control equipment installed in 1985, so spare parts are getting

hard to find.

Available support for this system was minimal from Mallinckrodt

& vendors.

6

Phase 1 (programming standard)

Why develop a Programming Standard.

There are basically two different control systems in our

facility. • Traditional DCS which has a programming standard.

• The rest are a combination of WonderWare, FactoryLink, RSView®(x),

Siemens, with various types of Allen-Bradley PLC based controls. No

common programming structure.

7

Phase 1 (programming standard)

No common programming

platform for the PAC based

systems.

Develop a programming standard for

future PAC systems that has the same

look & feel as our DCS systems.

For operators, the HMI graphics look

the same. This helps when moving

between manufacturing units with

different control systems.

For programming, the control module

structure is similar between the

DCS & PAC based systems.

8

Phase 1 (programming standard)

What was needed to develop this standard.

Partnered with Rockwell Automation Systems and Solutions

Business (SSB) & local Systems Integrators to help develop the

standard.

Functional Specification jointly developed with Rockwell

Automation (you have to know what you want)

Design Specification developed by Rockwell Automation based on

the Functional Specs.

Test Specification needed to validate the various control modules.

• Once validated, these Standard Modules could now be used on

multiple projects without re-validating.

Lessons learned

• It won’t be perfect the first time. You will miss something.

• We are on version 3 of our standard.

9

Phase 2 (the upgrade)

Scope

Partnered with Rockwell Automation Systems and Solutions

Business (SSB) to execute the project.

• This required very close collaboration of the project team.

Process Engineer, Automation Engineer, Validation, Rockwell

Automation Engineer. All the way through the entire project.

Replace a legacy Allen-Bradley PLC based Control System & HMI.

• Total PLC code re-write (~10K rungs) using S88 phase manager &

FactoryTalk® Batch. Utilized the Standard Control Modules created in

phase 1.

New HMI Graphics re-drawn from original HMI graphics.

• ~200 HMI screens (process graphics and Batch Record screens)

Re-created 60 Digital Batch Records and 6 Process &

Environmental reports utilizing Microsoft Reporting Services.

Utilize existing field I/O to reduce cutover time and risk.

10

Phase 2 (the upgrade)

Scope – continued

As-Built 64 Process Logic Descriptions (PLD’s) per existing PLC

code for accuracy. Then updated to a new Class-Based document

format.

• Managed only one document per operation per Class. Used Excel Mail

Merge function to populate the document with the correct devices. This

concept worked extremely well. A separate spreadsheet contained the unique

elements relative to all module classes in the application.

• 33 Unique Module Designs, + 31 Instances of those Designs. Class-based

documentation can be viewed and printed and be specific to the instance with

respect to Equipment and Devices.

• Class-based Methodology also served (i.e. focused us) to clearly identify all

differences between instances of the same class. (Fail open vs. Fail Closed

valves, extra instrumentation, etc.)

11

Phase 2 (the upgrade)

12

Process Logic Description Development Structure

Documentation Development:

• Each unit on Train 1 has its own Process Logic

Description in MSWord.

• MSExcel is used to provide the Mail Merge data to

each document. The Train 2 documents can be

automatically generated at the click of a button.

• Modifications to one Train are automatically

captured for the other Train.

• Support Systems require their own documents.

Phase 2

13

Mail Merge Demonstration

What’s Out, What’s Left, What’s New

14

Out

1 - PLC3

2 - PLC/5’s

DOS based prog. terminal

4 - FactoryLink HMI operator

terminals (WinNT)

Oracle Historian/Datalogger (WinNT Server)

Batch Record Server (WinNT Server)

3 - Alarm printers

Serial interface to Delta V

control system in adjacent

building

Left

All field I/O

•1771 Remote I/O

•1756 CLX I/O

•Flex I/O

•Profibus PA

•AS-i bus

•Ethernet VFD’s

Used existing I/O to

reduce cutover risk. Plan

to upgrade I/O in the

future.

What’s Out, What’s Left, What’s New

15

All Servers and Terminals are virtualized.

Servers

FactoryTalk® Batch

FactoryTalk® Historian Site

Edition

FactoryTalk® VantagePoint®

FactoryTalk® View Site Edition

(primary & secondary)

FactoryTalk® Directory

MS SQLSvr

Hardware

3 - HMI workstations

1 - Engineering terminal (all thin clients)

3 – ControlLogix® L74 controllers (all redundant)

13 – Ethernet EN2T comm. Modules

3 – Remote I/O Modules

(interface to existing Remote I/O)

Delta V Ethernet Virtual Interface

Module (VIM)

3 – Stratix 8000 Network switches

2 – Blade servers for our existing

VMWare

Batch Record/Graphics printer

Virtualization

Built upon our existing VMware infrastructure.

Two identical systems in two separate buildings.

Automatic backups of virtual images on periodic basis for disaster recovery.

Currently running 16 servers & 30 workstation images on VMWare.

And growing.

Eliminates the need to rebuild a computer due to hardware failure.

Saved ~$100K in computer cost by using virtualization.

16

Factory Acceptance Testing

Joint venture with Rockwell Automation and local Systems Integrators.

Accelerated schedule. Startup moved up by two months.

Performed on-site utilizing existing VMWare system while production unit

was still operating. Allowed us to test the throughput on our actual system.

Three months duration working two shifts.

Still finishing design/code development when FAT started (required

numerous code merge procedures to avoid invalidating FAT efforts).

Leverage FAT testing for Validation. (did not re-test sequencing functions

during shutdown OQ)

17

Qualification & Startup (IQ/OQ)

Installation Qualification (IQ)

Old controls removed and new controls installed in 4 days ready for I/O

checkout.

Round the clock testing.

2 weeks duration working two shifts. Every device manipulated.

• 1000 digital, 500 Analog.

Operational Qualification (OQ)

Introduction of utility chemicals with water as the product

Round the clock coverage

2 weeks duration running multiple batches

18

Qualification & Startup (PQ)

Process Qualification (PQ)

Introduction of actual product.

Five batches were run initially.

• All five passed quality requirements.

Best of all:

No Safety issues

No Environmental issues

No lost batches

19

What went well

20

What went well

Programming Standard (HMI & controllers).

Excellent documentation for FAT testing and validation.

Good in-depth reviews of the Process Logic Descriptions.

VMWare & Thin Client usage.

Utilize our existing virtualization infrastructure to lower control system

maintenance cost & downtime.

FAT performed on-site and on our existing VMWare system. We can

then see how the system will perform before startup.

Get your people involved. Process Engineers, Facilitators and

Operators all know the process the best.

Near-duplicate Virtual Environment at Rockwell Automation Office used

for development, functional testing, mid-project demonstrations

(Webex’s), troubleshooting, and long term support .

Lessons learned

21

Lessons learned

You can’t have enough help.

Get involved earlier in the details of the program code to better

understand its structure by the time the FAT & Startup begins.

Validation protocol reviews could have been better coordinated.

Needed a better understanding of the VMWare infrastructure in our

facility.

Get a better handle on the Ethernet Network Infrastructure in our facility.

Get your Systems Integrator(s) involved with all aspects of the project…

Network Architecture and Configuration, other IS-related functions.

Going Forward

22

Future State

The Programming Standard has been used on 4 upgrade projects to

date, with another grass roots project set to begin in Q3/2014.

Virtualization will only grow in the future as our current control systems

become outdated. This has greatly reduced our computer maintenance

and infrastructure cost.

Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.

We care what you think!

On the mobile app:

1. Locate session using

Schedule or Agenda Builder

2. Click on the thumbs up icon on

the lower right corner of the

session detail

3. Complete survey

4. Click the Submit Form button

23

Please take a couple minutes to complete a quick session survey to tell us how we’re doing.

2

3

4

1

Thank you!!

Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.

PUBLIC INFORMATION

Thank you for participating!

Please remember to tidy up your work area for the next session.

We want your feedback! Please complete the session survey!