Post on 22-Dec-2015
Session ID 2853 1
Presented by: Brent Maynard, Myke WattsMarshall University
April 9, 2013
Session ID 2853
Banner Virtualization
How Virtualization adds flexibility to our production environment, and enables new opportunities
Session ID 2853 2
Session Rules of Etiquette
• Please turn off your cell phone/pager
• If you must leave the session early, please do so as discreetly as possible
• Please avoid side conversation during the session
Thank you for your cooperation!
Session ID 2853 3
Introduction
• In this session, we hope to show you the benefits we have gained by moving from a physical 32bit architecture to a virtualized 64bit architecture.
Session ID 2853 4
Agenda
• Where we started• Why a virtualized Environment• 64bit benefits• Migration – Source code and database• Infrastructure • Cloning• Performance, Reliability, and Disaster
Planning
Session ID 2853 6
Where we are – all virtualized
• Two transactional database servers• Two ODS/EDW database servers• Four Banner instances• Four Middleware servers (INB, SSB, etc.)• High availability• Increased performance• Business continuity
Session ID 2853 8
Where we started
SSB INB
BANNER
BANAUX
BANNER
BANPROD
BANNER
BANSTAGE
PHYSICAL (32 bit)
ODS/EDW ODS/EDW
warehouse warehouse
COGNOSARGOS
COGNOSARGOS
Session ID 2853 9
Background – 32bit Banner
• Redhat Enterprise RHEL5 32 bit• Dell R710 dual socket quad-core with
12GBytes memory• Dell Equallogic 6510x storage array• 1.5 TBytes allocated storage• Oracle 10g Database and Application
Services• Typically we have ~250 concurrent Banner
Forms users
Session ID 2853 10
Background - continued
• Due to multi-core expansion over the past few years, we decided to virtualize our INB/SSB Oracle Application Services because of Oracle OAS licensing constraints 18 months ago.
VMware ESX hostDual socket, quad core
MUSSBDell dual
socket dual core
MUINBDell dual
socket dual core
Before
Vmware Client
MUSSB4 cpus
Now
Vmware Client
MUINB4 cpus
Session ID 2853 11
Background - continued
• We have migrated to the 11g Oracle Fusion infrastructure• Hopefully by the time of this presentation, Java 7 support.
OFM Classic DomainINB/SSB/DWproduction
OFM Basic DomainBEP/WF/BRM
production
OFM Classic DomainINB/SSB/DW
preprod
OFM Basic DomainBEP/WF/BRM
preprod
ESX Host
Session ID 2853 13
Why did Marshall University Virtualize?
• $$$$$$– Reduced server expenditures– Reduced power requirements– Reduced cooling requirements– Service availability– Flexibility
Session ID 2853 14
Why did Marshall University Virtualize?
• At Marshall, the majority of our core services are now running in the ESX environment.– Exchange– Sharepoint– Web (www.marshall.edu)– Banner (Workflow, BRM, DegreeWorks,
ODS/EDW)
Session ID 2853 16
Why 64bit?
• RHEL5 removed highmem constructs that RHEL4 had allowed us to use to grant Oracle large chunks of memory.
• RHEL5 32bit caused us some issues when our pool space was consumed during heavy use periods.
• With 64bit linux and Oracle, we can now grant Oracle large chunks of memory (7.5 Gbytes) .
Session ID 2853 18
Migration
• Install Oracle 11gR2 64 bit DB server• makefiles• Cobol• library paths• rsync
Session ID 2853 19
Migration – 11g database
• Install Oracle 11gR2 Enterprise Edition• Allow Oracle Support to apply the latest
patch set• Configure to connect to Grid Control• Confirm the latest CPU patch was applied• Allow the installer to create a test
database• Install Oracle 11g Client (full client)
Session ID 2853 20
Migration – makefiles
• Download the Banner 8 makefiles from Ellucian and choose the file that is specific to your OS
• FAQ CMS-1361 Banner Example Makefiles
Session ID 2853 21
Migration – Cobol
• The Pro*Cobol pre-compiler is not included with the Oracle 11g Enterprise Edition install. It is included with the 11g Client and must be installed separately
• $COBOL=/opt/FJSVcbl• $COBOL=/opt/FJSVcbl64
Session ID 2853 22
Migration – rsync
• To move the file system from the 32bit environment to the 64 bit environment, we used the linux rsync command.
• It takes approximately 4 hours to sync the file system.
• rsync benefits– Maintains file dates– Maintains file permissions and ownership
• File system ACL’s do NOT migrate, so those will need to be applied manually if you have any set.
Session ID 2853 23
Migration – Database
• Create new production database (empty)• Oracle export/import• Duration• Post import cleanup• BAN_ roles replaced by USR_ roles• Database ACL’s (workflow, smtp, http)
Session ID 2853 24
Migration – Database
• Use DBCA to create a new Banner database (general purpose)
• Give it the same DB name as the previous Banner database
• Configure the new database to connect to Enterprise Manager Grid Control
• Configure a listener
• Set appropriate locations for datafiles, logfiles, control files
• Choose the correct Unicode character set AL32UTF8.
Session ID 2853 25
Migration – Database
• The Oracle 11g Grid Control does not support database cloning from 10g 32bit to 11g 64bit.
• Use either the Oracle import/export or datapump utilities.
• DegreeWorks issues– Even though the export was successful with no errors, the
subsequent import has missing DegreeWorks objects.
– We had to manually create the DDL to recreate the missing DegreeWorks objects.
Session ID 2853 26
Migration – Database
• For our instance, it took approximately 11 hours to export the database, copy the export file to the new server, and import the file into the new database.
• (Fee Assessment Audit Table)
Session ID 2853 27
Migration – Database
• Oracle 10.2.0.5 and 11g no longer allows password protected roles such as BAN_DEFAULT_M to be granted as default roles
• BAN_DEFAULT roles are replaced by USR_DEFAULT roles
• Users in your current database with BAN_DEFAULT roles will have to be granted the corresponding USR_DEFAULT role
Session ID 2853 28
Migration – Database
• Ellucian supplies three scripts to aid in the role conversion
gcnvconnectrole.sql
gcnvconnectrole_ban_default_m.sql
gcnvconnectrole_ban_default_q.sql
• These scripts will create the new roles and apply them to users either by adding the new role or replacing the existing role.
• Ellucian FAQ’s:1-XFTOI5
1-H4TMPD
Session ID 2853 29
Migration – Database
• Oracle 11g requires ACL’s to be added for DegreeWorks, Workflow, and any custom apps that perform HTTP or SMTP calls from the database. FAQ# 1-648TQ0
• ACL_and_Banner_11g_database.pdf
Session ID 2853 31
Infrastructure
• We have an excellent infrastructure team!• VMware templates have been created.• Flexible storage solutions using the Dell
Equallogic storage array.• OPSCODE Chef provisioning services
– Server configuration (iptables, users, httpd, profiles, groups, etc.)
– Configuration management
• A 64bit RHEL5/6 server can be provisioned, configured, and booted in around 15 minutes.
Session ID 2853 32
Infrastructure – Grid control
• We use Oracle 11g Grid Control to manage our Oracle database infrastructure.– Backups– Alert Monitoring– Database Cloning– Performance Monitoring– Regular Maintenance
Session ID 2853 34
Infrastructure – Grid control
• For our Banner Environment we have multiple instances running.– BANPROD– BANAUX1 Clone of production for Research Corporation testing– BANAUX2 Clone of production for University testing.– BANSTAGE Staging environment for new releases
• BANAUX1 and BANAUX2 share a common code tree
Session ID 2853 35
VMware – current versions
• vSphere Software– Vmware vCenter Server v5– Vmware ESXi v5 Hypervisor
• vSphere Features– vSphere HA – clustering ability to hot migrate VM
from one physical server to another– vSphere DRS – ability to balance performance
across cluster
• vCenter Site Recovery Manager
Session ID 2853 36
VMware Hardware Architecture
• 10 x Dell r710 Aggregate Specs– 108 CPU (287 GHz, mix of Nahalem and Westmere) – 1.03 TB 1333Mhz Memory
• 48 TB iSCSI storage (3 tiers)– Equallogic 6010x Hybrid SSD/SAS 15k 10GBe (8 x SSD, 8 x 15k SAS)– Equallogic 6510x 10k 10Gbe (48 x 10k SAS)– Equallogic 6510e 7.2l 10Gbe (48 x 7.2k SATA)– 4 x Dell 8024f SFP+
• Networking (per server)– 2 x 1Gbe – VMkernel & vMotion– 6 x 1Gbe – Virtual machine data networking– 2 x 10Gbe – iSCSI network storage
Session ID 2853 42
VMware and SAN cloning
• With VMware, we can now easily clone– Servers– Environments
• With Equallogic, we can clone data LUNs
Session ID 2853 46
VMware cluster
• VMware server farm– No difference in performance compared to standalone server– IO throughput is superb with multiple 10GBytes i-scsi
adapters– vMotion and dynamic load balancing among cluster members– Cloud services for DR contingency
• We are talking with cloud services vendors to pilot a DR hot-site clone of our production environment.
• Sister Institutions within the state of WV we share a 10Gbit fiber ring.
– Rack swap– VMware Site Recovery Manager– Equallogic SYNCREP (byte level SAN replication)
Session ID 2853 47
Summary
• Server virtualization is working very well for Marshall University.
• We have reduced infrastructure costs,• Gained a higher level of service for
customers.• Increased flexibility to respond to customer
needs (multiple Banner instances),• And created a viable disaster recovery
solution.
Session ID 2853 49
Thank You!
• Brent Maynard – brent.maynard@marshall.edu– (304) 696-2591
• Myke Watts – iwatts@marshall.edu– (304) 696-3921
• Presentation available• http://webpages.marshall.edu/~maynard169/live2853.pptx
.
© 2013 Ellucian. All rights reserved.