E550I Bayer Case Study: Manufacturing Excellence in the

Biotech Industry David Kavanaugh, PCS Engineer Berkeley, CA

ABB Automation and Power World Presentation 2012

David Kavanaugh

• Process Control System Engineer for Bayer HealthCare, Berkeley CA

• PCS since 1978, when “huff and puff” was still King

• Oil, Gas, Chemical, Food, Textile, Nuclear, Pulp/Paper, Power,

Pharmaceutical and others I don’t remember

• First ABB (Taylor MOD 300) exposure 1989 at co-gen plant in Rocklin,

CA

• ABB Resident Engineer at Bayer 1995

• Bayer PCS Engineer since 2001

• Current duties: Data and Process Visualization and ABB Support

backup

Bayer, Berkeley CA

Kogenate: a Factor VIII blood clotting agent for

persons with hemophilia, a blood clotting disorder

Bayer and ABB: Twenty-four years of Partnership

- 1988 First MOD 300 System: Building 55

- 1989 Two MOD 300 Systems: Buildings 52 & 5A

- 1992 Two MOD 300 Systems: Buildings 57 & 49

- 1994 MOD 300: Building 60

- 2001 B55 converted to Process Portal B

- 2002 B60 converted to Process Portal B

- 2003 New PPB system for new Building 81

- 2004 Buildings 49 and 5A converted to PPB

- 2005 800xA Site Wide Historian

- 2005 New 800xA system for new Building 66

- 2008 B60 and B81 evolving to 800xA and Batch

- 2009 Enterprise Connectivity Solution for Manufacturing Execution Systems (Electronic Bach Records)

- 2010- 800M Controllers and Increased Batch Implementation

- 2012- Evolved B55 Process Portal B system to 800xA 5.1 with PG2 graphics

EFFICIENCY & FLEXIBILITY is now

a Strategic Requirement for Plant Automation

EFFICIENCY & FLEXIBILITY is now

a Strategic Requirement for Plant Automation

Systems Yesterday

Low level of Automation (no closed loop, advanced control)

Poor standardization

Automated sequences were “hard coded” (not flexible)

No, or difficult documentation of programming code.

No, or difficult, interconnection between different manufacturers

Hard or impossible to automatically get data from a system

Systems Today

Process analyzers and advanced control implemented

Use of OPC and standard 3rd party drivers

Electronic Batch records with recipes selected through MES/ECS systems

Self-documenting tools and auditing

Systems integrated in a site wide Network among different Plants/ Utilities/ Buildings and with IT (MES Electronic BPR, integrated Paging, Remote Maintenance, Reporting, ...)

Data automatically transmitted, web services

How Did We Get There

Process Control System Lifecycle Options

Risks Rewards

• Maintain Lack of New Parts No Capital Expenditures

Higher Maintenance Costs Familiarity

Unable to meet strategic functionalities No new training required

• Rip & Replace Production down time New Features

(Vertical*) Large capital expenditure Longevity

Increase training Lower maintenance costs

• Evolve Increased time to completion New features

(Phased*) Higher costs over time Flexibility

Schedule Less time pressure

*ARC Strategies: The Control System Migration Survival Guide, March 2010, Larry O'Brien & Dave Woll

We Chose to Evolve

How Will We Evolve?

• Changes are occurring through concurrent multiple projects: MES,

DCS upgrades, batch implementation and others

• Learn the new skills required for new technologies.

• Add new functionalities to our systems (connections to other

systems, remote desktop, advanced control, etc.)

• Hardware and software changes, standardize across systems

• Partner with internal and external suppliers

Automation Projects

• New site-wide DCS platform to integrate existing DCS systems. Upgrade

to latest 800xA version will allow the integration.

• Integration to MES for Electronic Batch Record following the ISA S95

standard

• Will add new small scalable controllers for on-skid installation (800M

Controllers) while utilizing existing controllers/logic for remainder of plant

• Flexible integration connector (ECS) will also allow future integration

with other Control systems (BAS, SCADA) and IT systems (cMMS, SAP,

LMS, ..)

• Open standard automation interfaces (OPC, Fieldbus) to be less

dependent on a single vendor and to be ready for PAT integration

Automation Projects

• Introduction of ISA S88 Batch in API manufacturing will prepare the

basis for Berkeley to became a multiple-products site, through flexible

control-recipes management (new products)

• Object-based development (reusable library objects) will allow

streamline implementation and validation and will simplify system

documentation (Bayer “Standard Libraries)

• Paging functions and equipment efficiency (OEE) will occur through

Asset Optimization

• New Historian to connect all PCS systems (web services reporting, data

analysis tools,…)

Site Automation Network Infrastructure

Local DCS SystemsBatch and Continuous Controls

Historical data collection,

Remote Desktop,

Alarms, Paging

Corporate IT Network

ABB DCS

ABB DCS

ABB DCS

ABB DCS

ABB DCS

ABB DCS

DMZ

ECS BPR Connector

DCS web services

Backups services

Antivirus services

Site DCS Historian

Site BAS Historian

Site REES Historian

Central ServicesElectronic Batch Records

Historical data collection,

Backups, Antivirus

Web Reporting

Other Local Systems

Labs Supervision REES SCADA

Packaging Supervision SCADA

Fermentors Fermworks SCADA

WFI Supervision SCADA

Particle Monitoring System

Utilities Supervision SCADA

Electrical Distribution Monitoring

BAS Building Automation Systems

IT firewall

PCS firewall

Lessons Learned Through Evolution

• Be open to changes in the schedule (have alternative

“next steps”)

• Partner with all stake holders (suppliers, internal

suppliers, and especially your end users)

• Keep an Open Mind, be receptive to new ideas and

methods

• The systems are becoming more of an IT based system,

requiring IT skills

Schedule Problems

Challenges to Meeting Schedule

• Production schedule requires flexibility to work around

last minute changes, stay informed

• PCS “up time” is different, might require a phased

approach

• Training of operators in new HMI use

• Un-expected changes to design

• Staffing and training

Partnering Issues

Partnering

• Identify all stake holders

• Solicit input with several review cycles through-

out the entire process

• Leverage the experience your partners have

(what a graphic should look like, how systems

can connect, what type of computers, etc.)

New Ideas

Incorporating New Ideas

• New systems = New ways

• Leverage partners experience

• Design for “Openness” (flexibility, connectivity)

• Solicit input/ideas from others (Trade Shows,

trade magazines, other industries)

• Implementation through Lab Development

systems

PCS Meets IT

A “New” IT System/Infrastructure

• We now use “standard” IT hardware/software

• Interconnection between the PCS and IT worlds must

follow IT rules

• We need the same “services” (anti-virus, OS patches,

ABB patches, Third party patches, performance

monitoring, etc.)

• We “talk” the same “talk”: TCP/IP, firewall, switches,

routes, permissions, etc.)

The Challenge of Using Your IT Department

Challenges:

• The IT department knows Windows: Yes they do, but they may not know process

control and the special requirements need for it.

• The IT department can provide on-call coverage: When we investigated the on-call

procedure we found that the “best” response time available was 4 hours.

• The IT department has structure: Yes, but do they report up through the same chain of

command for escalation of issues?

• Costs: Original proposal for full time support was $175/month per server. 200 servers x

$175/mon. = $420,000/year

Solutions:

• Work on the IT departments training: and the Engineering department’s training of IT

technology.

• Supplement with Engineering on-call: engineers love to help.

• Service Level Agreements: specifically outline responsibilities and escalation of issues

• Purchase warranties from vendors: for on-site support, have well defined SLAs

• Align the evolution schedule with Production Planning to work within their

production schedule and implement new equipment while concurrently

running old equipment, continued production and “live” training

• Investigate ways to run old and new systems in parallel

• Involve end users from the beginning i.e. use operators to design and

build or approve graphics

• Work with ABB or System Integrator to have a defined method on how

to implement required changes with little or no impact to schedule

• Have “back-ups” for essential personnel and invest in “pre-

implementation” training of critical personnel

Mitigating the Challenges

Berkeley Site Automation Strategy, an Integrated Vision

HVAC PACK UTIL API FPM CMF

AUTOMATION

MES

ERP SAP

ELE

EQUIPMENT

QC GBD

Plant Automation Network

SCADA Rees

Centron

Wireless

Historian

DCS ABB 800xA - Batch

800M – AC460 – Rockwell PLC

(PAT integration – Profibus)

Historian

SCADA GE iFix

Rockwell PLC

Siemens PLC

Historian

BAS Siemens

Apogee

Historian

SCADA Siemens

WinPM

Siemens PLC

Historian

IT Corporate Network

eBPR ABB/Werum

LIMS cMMS EDMS

Benefits • Flexible integration connectors in place for integration with future Control and IT

systems (cMMS, INtools, LMS, ..)

• Open automation interfaces (OPC, Fieldbus) and Standards (ISA S88, ISA S95) allow

us to be prepared for future expansion with connections to any type of system

• The new platforms/infrastructure is the basis for other services such as a site-

Historian database, web services for plant access and reporting, data analysis tools,

etc

• Use of “standard” IT equipment and standards decreased time for installation and

allows for sharing of knowledge with local IT “experts” including Corporate

• Allowed the introduction of Web based access to production data reducing costs and

capital investment

• Remote engineering access to the Production systems reducing time to implement

new control strategies and time to support automation problems

What’s Next?

• Continued “evolution” of the PCS systems for flexibility and

connectivity

• Virtualization of PCS systems

• New Site Wide Historian for all PCS systems

• New Operator interfaces/technology (tablets, PDAs/Smartphones,

cellular, ?)

• Remote connectivity/access through Web based applications (Cloud

Computing, “WIKI” type search capabilities)

• Tighter integration between PCS and IT groups

Questions?

backup slides

BAS

PLC

PAT

Instrument

DCS

SCADA

SAP

SRM

ERP

CRM

cMMS

EDMS

LIMS TD

PP

TOPAS

SPE

eBPR

WMS

MM

Puzzling Array of Technologies

Automation

Layer

MES Layer

Enterprise

Layer

IT

Eng

ine

ering

Systems Distributed in Layers (ISA-95 Standard)

AM ?

B60: Completion of 800xA +Batch +800M

(common to all systems)

RTDS #2B w/

PPB 1.2 Client Node 2900

W 2KS Mod Connect

2.3/0

RTDS #1A

w/ PPB 1.2 Client

Node 1800 W 2KS

Mod Connect

2.3/0

(DCN 14.6/7)

AdvaBuild - ES

3.3.02p2 Node 100

W 2KS w/ MED & TCL Bldr

RTDS #2A

w/ PPB 1.2 Client

Node 1900 W 2KS

Mod Connect

2.3/0

Plant w ide

History

Consolidation

W 2KS

DNS #1 W 2KS

DNS #2 W 2KS

(32) PPB 1.2 Clients

(37) clients total

EH 3.3/2

Node 1600 W 2KS

EH 3.3/2

Node 1700 W 2KS

PPB 1.2 Client & Config Srvr

RTDS #1B w/

PPB 1.2 Client Node 2800

W 2KS Mod Connect

2.3/0

MOD OPC

Server 1 Node 600

W 2KS

MOD OPC

Server 2 Node 700

W 2KS

Multibus

DDE

(300)

Plant A 460 Controller

Node 0B00

(3) 1:1 Rdnt

CS01 CS02 CS03 CS05

Utilities 460 Controller

Node 0F00

(3) 1:1 Rdnt

Plant A 460 Controller

Node 0C00

(3) 1:1 Rdnt

Plant B 460 Controller

Node 0D00

(3) 1:1 Rdnt

Plant B 460 Controller

Node 0E00

(3) 1:1 Rdnt

CS04

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

TRIO LAN

Redundant Ethernet TCP/IP

TRIO LAN

TRIO LAN

TRIO LAN

New

KG-PF

Skids

Information Sharing

• Electronic Batch Records

- Pre-built solution: Enterprise Connectivity Solution (ECS)

- Stand-alone : batches run regardless of connection to the MES

system

• ABB Smart Client

- Allows for “safe” access to DCS information

- Large Bang for the Buck

- Low system resource overhead

- Easy to configure

Flexible, Scalable, Controls: 800M

• 800M uses current supported programming standards

• Easily integrated into the 800xA system

• Allows for easy “model” type programming

• Hardware costs were comparable

• Skid Integrators were eager to partner and learn new

hardware.

Bayer Berkeley History

• Cutter founded in 1897 (Fresno, CA)

- Black Leg vaccine for cattle

• Cutter moved to Berkeley, CA in 1903

- Small pox Vaccination at 1906 earth quake

- Allergens developed 1927 (Poison Oaks and Ivy)

- Tetanus (Horse Serum), 1930

- Major producer of penicillin, 1943

- Plasma fractionation acc. to Cohn during WW 2

- Mosquito repellents, snake bite kits, 1950’s

• Plasma Fractionation plant built 1967

• Konyne: Hemophilia B, 1969

• Koate: Hemophilia A, 1974

• Cutter absorbed by Miles Laboratories, 1978

• Prolastin: Congenital Emphysema, 1987

• Kogenate: 1993

• Miles Laboratories became Bayer in 1994

Efficient Operations: Automation, Flexibility, Integration

Berkeley Site Yesterday

Low level of Automation (most

operations are executed manually)

Low flexibility (our automated

sequences were mostly “hardcoded”)

Local, stand alone, isolated Systems

in each Plant /Building not allowing for

data exchange and external services

Berkeley Site Today

AUTOMATION

New platforms will allow automation of manual processes

SCADA supervision

FLEXIBILITY

Pre-built and pre-validated softwarew libraries

Batch control: flexible use of multiple “recipes”

Scalable controllers in process areas

SCADA supervision in Utilities and

Packaging on top of standard PLC

INTEGRATION

All PCS systems integrated in a sitewide Network among different Plants & Buildings

Integration between PCS and IT (MES Electronic BPR)

Integrated Paging of Alarms

Remote System Maintenance

Data mining, System reports, etc

Evolve 1. Why

• A need to change our systems while maintaining production, reliability, increased efficiency, access and accountability

2. How

• Partner with PCS vendors for system flexibility/expandability

• Partner with Production to minimize conflict with product delivery

• Co-ordinate access and data flow with Corporate IT

• Develop a robust and secure network infrastructure

3. What

• A system that meets all user’s data requirements, Production, Data Miners, Engineers, for robustness and accessibility

• Secure, complete access of information to all users

• Traceability of code changes and information flow

MES-DCS Implementation: Basic Connectivity

Execution on DCS level

Master BPR

Unit procedure 3

Start OP 1 OP 2

OP 3

End

BF 1 BF 3 (part recipe)

BF 2 BF 4

MES

Automatic run per Unit

Control Recipe

DCS BATCH

Unit procedure 4

Start OP 1 OP 2

OP 3

End

batch_ID

recipe_ID

start unit_ID

EBR relevant

data

(end of Unit

procedure)

EBR relevant

data

batch_ID

recipe_ID

start unit_ID