Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
Brad McCuskerSoftware Concepts International, LLC57 Technology Way, Suite 2W1Nashua, NH 03060, USA
Phone: 603-879-9022
e-mail: [email protected]
www.sciinc.com
Proven results by applying the
best practices in Computer
Systems Management
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Financial Services – can we do it?
� Latest & Greatest as long as nothing changes
� VAX to Alpha with 90% of the pieces
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
Managing OpenVMS systems and databases requiring the highest
levels of performance and availability – worldwide
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
4
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
5
� Replace aging DS-25 & MSA
� Virtualize the platform
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� DS25 with 1GHz processor
� Concern over emulator performance
� Could we satisfy the extended periods of high CPU?
� In 2010 – told customer low confidence of success
6
050
100150200250
9:4
1:2
4 …
10
:16
:33
…
10
:51
:43
…
11
:26
:52
…
12
:02
:02
…
12
:37
:11
…
1:1
2:2
1 P
M
1:4
7:3
0 P
M
2:2
2:4
0 …
2:5
7:4
9 …
3:3
2:5
9 …
4:0
8:0
8 …
4:4
3:1
7 P
M
5:1
8:2
7 …
5:5
3:3
6 …
6:2
8:4
6 …
7:0
3:5
5 P
M
7:3
9:0
5 …
8:1
4:1
4 …
8:4
9:2
4 …
9:2
4:3
3 …
9:5
9:4
3 …
10
:34
:52
…
11
:10
:02
…
11
:45
:11
…
12
:20
:21
…
12
:55
:30
…
1:3
0:4
0 …
2:0
5:4
9 …
2:4
0:5
9 …
3:1
6:0
8 …
3:5
1:1
8 …
4:2
6:2
7 …
5:0
1:3
6 …
5:3
6:4
6 …
6:1
1:5
5 …
6:4
7:0
5 …
7:2
2:1
4 …
7:5
7:2
4 …
8:3
2:3
3 …
9:0
7:4
3 …
% CPU Idle Time
%Idle
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Existing� DS-25, 2P (EV68 1.0GHz), 10GB
� OpenVMS 7.3-2
� MSA 1500, 3 volumes, ~600GB
� Proposed� Dell PowerEdge R630
� 2 - E5-2637 v3 3.5GHz,4C Processors, 128GB
� VMware ESXi 6.0
� Windows 2012
� Digi AnywhereUSB (for CHARON key)
7
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Boot emulator with temporary VMS
� Network transfer of savesets
� Shut down Alpha
� I hated to bring this one down:
� Reconfigure, reboot emulator
� Overall, uneventful migration
8
AXP>show system /noproc
OpenVMS V7.3-2 on node XXXX 16-MAY-2016 12:10:13.08 Uptime 1317 15:15:03
AXP>
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Winsock 10035 error when trying to connect from clients
� Google, etc. indicated firewall problem
� Network team disabled a firewall somewhere
� Problem solved
� BZZT!
� Many days later, same problem, firewall still disabled
� Lots of wasted time chasing phantom firewall issues.
� Ultimately resolved by executing @CNXSVR again
9
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� CPU:
� Our experience with ProLiants is CHARON/VMWare should be around 2277 VUPs
� Dell: 94% as fast
� Followed DELL guidelines for high performance, simply couldn’t make it faster
10
CHARON: Approximate VUPs Rating : 2149.3 (min: 2121.0 max: 2167.0)
DS25: Approximate VUPs Rating : 2542.8 (min: 2538.0 max: 2547.0)
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Initial results: batch jobs runabout 25% slower
� 50-70% CPU, 1500-2000 DIO
� Tuning:SET RMS/BLO=127/BUF=8/SEQ and
SET RMS/BUF=128/IND
SET RMS_SEQFILE_WBH =1
SET RMS_DEFAULT/QUERY_LOCK=DISABLE
�Batch job performance went to 90%
11
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Alpha: MSA 1500
� 15K SAS drives
� CHARON configured with “raw SCSI” devices
� VMware .vmdk actually
� Started with NetApp SAN
� 1 GB Ethernet, 15K SAS drives
� Next: DellSCv2020 ISCSI 10GbE Dual Controller
� 1GB switches, etc. 10K SAS drives
� NetApp was temporary use only
� Separate project replacing NetApp with Dell
12
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� I/O with NetApp: Faster than MSA
� I/O with Dell: Slower than the MSA
� Customer was concerned
� Proposed using Windows I/O caching
� Risk: I/Os in Windows cache could be lost
� Customer willing to accept that risk
13
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
14
0
2000
4000
6000
8000
10000
12000
14000
16000
1 2 4 8 16 32 64
Xfer/sec
Q Depth
256 block Random Writes
Container with windows cache DS-25 & MSA VMDK Raw SCSI
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Switch to Windows cache resulted in applications running at 95% of the Alpha
� Acceptable results, customer happy!
15
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Current procedure:
� Interactively run a procedure
� Walk into the computer room and insert/swap tapes.
� Every day!
� Most tapes kept onsite – Friday tapes sent offsite.
� New process:
� Backup to virtual tape on VMWare guest
� Virtual tape is a windows container file
� Virtual machine gets backed up to local repository
� Copy of the virtual machine is sent to the cloud
16
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Fully automated, VMS Batch
� No more physical tapes
17
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
18
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Alpha � Integrity
� EVA � 3PAR
� Custom code migration
� Servers hosted/managed by 3rd party with limited modern VMS knowledge
� Move new systems in-house
19
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� GS1280, 2 partitions (12P/4P)
� EVA5000, dual controller, Business Backup� 168 36GB 18K drives, RAID 0+1
� BASIC, C, COBOL, Datatrieve, DECForms, CMS SNA/LU6.2
� Oracle Rdb/CDD
� IBM MQ Series
� Xentis, AIG UDMS (report generation)
� CA PA(VPA), DC
� Not available IA64: Raxco Control, FaxSr, Eigen Utilities, SuperSqz
20
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� 2x HP Integrity BL870 servers� 4 x I2 9350 (1.73GHZ/4C), 64GB
� C7000 enclosure
� 3PAR 7200C – 2 nodes
� 24 x 920GB SSD
� Test/Dev: RX2800, 2P I2 9340 (1.66GHz)/4C
� Software: Same as Alpha, except:� NDS/Compufax, Peek/Spy, xxxx?
21
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� 95% BASIC, rest is C, COBOL, Macro
� TOLAS Run time libraries (owned sources)
� RMS files for data
22
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Provided loaner IA64 server for intial porting POC
� Configured dev/test in NJ data center� Difficulties with ILO address
� Migrated from loaner� Storage in NJ????
� Customer moved dev/test and 3 PAR to CT data center
� Assist with planning for enclosure switching, 3PAR configuration
� Configured C7000 and blades in CT data center
� HP configured 3PAR� HP ‘expert’ really wasn’t
23
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Old system, from VMS, � Imitated an EVA based snapshot of disk to be backed
up
� Present snapshot via SSSU CLI to VMS
� VMS Backup to tape used against privately mounted snaps
� New system,� Plan was to Use 3 PAR Virtual Copy from VMS using
the Python-3parclient toolset
� Didn’t work out
� Used SSH to 3PAR to execute 3PAR commands. 3PAR snaps, then used VSM backup on private volumes
24
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Significant work to emulate current installation while taking advantage new technology
� Different customer team members lacked “big picture” knowledge or view
� SCI provided big picture oversight
� Systems reside in a hosting company’s Data Center
� Some of the network was controlled by the hosting
company
� Hosting personnel’s first HP blade enclosure
25
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Customer handled all testing
� Didn’t do enough
� “We tested what we thought was important”
� Didn’t test things that were difficult to replicate
� FTP, SFTP, FAX, SNA
� WebsphereMQ
26
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Delayed due to testing issues.
� SCI engaged to help troubleshoot
� Ultimately successful, couple week late
� SCI retained to provide Managed Services.
27
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Financial sector customer wishes to upgrade critical production environment to current supported VSI OpenVMS and Oracle Rdb releases
� SCI engaged to perform production system and database upgrades and provide post-upgrade analysis and support
� Engineer forced to do international travel
29
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Audit 10 months ago identified unsupported database and operating system in production
� Planning for upgrades failed to achieve required priority
� Production upgrade rush schedule� No test/development upgrade (more about this later)
� Limited testing
� No application changes – only VMS and Rdb and related products / options
30
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Upgrade process tested on CHARON clone of production
� Upgrades and database converts went well
� ‘Smoke tests’ for a few hours
� User test for a day
31
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Upgraded production system Sunday morning early� Branches open at around 10:00
� Upgrades and installs went well
� Database converts failed due to logical name complexities (process, group and system) from different upgrade account (SYSTEM vs application)� Quick restore of database
� ‘Smoke tests’ for an hour before full production
32
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Upgraded
� OpenVMS V7.3-1 to V8.4-2L2
� Rdb V7.1-01 to Rdb V7.3-21
� System and database analysis and suggested tuning
� Users report improved performance
33
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Average can further be significantly improved with database tuning
34
Node: ACCPQ (1/1/16) Oracle Rdb V7.3-210 Perf. Monitor 25-JUL-2017 09:49:42.39
Rate: 3.00 Seconds Transaction Duration (Total) Elapsed: 08:52:23.52
Page: 1 of 1 DISK$SBS:[STFEXP.MAE]SBS.RDB;2 Mode: Online
--------------------------------------------------------------------------------
Total transaction count: 58829
Seconds Tx.Count: % #Complete: % #Incomplete: %
0-< 1: 57675 98% 57675 98% 1154 2% <-avg=0.167410 95%=0.20
1-< 2: 462 0% 58137 98% 692 2%
2-< 3: 235 0% 58372 99% 457 1%
3-< 4: 194 0% 58566 99% 263 1%
4-< 5: 79 0% 58645 99% 184 1%
5-< 6: 25 0% 58670 99% 159 1%
6-< 7: 8 0% 58678 99% 151 1%
7-< 8: 15 0% 58693 99% 136 1%
8-< 9: 16 0% 58709 99% 120 1%
9-<10: 1 0% 58710 99% 119 1%
10+++: 119 0% 58829 100% 0 0%
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017 35
OLTP STF SBS
Database
AIJ
JCC
LogMinerLoaderRdb Continuous
LogMiner
• Near real time CDC + ETL• OLTP data available for read
access shortly after commit• Change data collected from
live AIJ files – no database access
• Zero / minimal OLTP impact• Offload read-only access to
SQL Server• Via JCC LogMinerLoader
“Mirror” SQL
Server DatabaseAIJ
AIJAIJ
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� Required upgrade to Rdb 73 and VMS 842L2
� Mandatory in order to enable restore of production
databases to test/development environment
� Optional migration to CharonAXP-AXP on VMware to improve performance and reliability
36
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� More processing units
� Increase system headroom
� Permit additional jobs to run in parallel
� Does not improve single-job performance; increases
system capacity
� Current system 2 emulated processors
� Charon ES40 Emulator supports 4 processors
� Requires additional CPU licenses
37
Copyright © 2016
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.29 September 2017
� [TCPIP$DHCP]TCPIP$DHCP_RUN.LOG was the busiest file on the system disk
� Full of error messages
� DHCP server is not required nor desired on this system
� The DHCP Server has been disabled
38
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
Brad
Brad McCusker
www.sciinc.com
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� Located in Nashua, NH (USA)
� 30 years in business supporting OpenVMS!
� International reputation as leading provider of
� Managed services for OpenVMS & databases
� OpenVMS performance and consulting services
� Proven global track record
� Actively managing 100s of systems and databases
� At many sites 24x365 since 1995
Copyright 2017
SCI, LLC, Nashua, NH USA
Software Concepts International, LLC.World Class Managed Services For OpenVMS
� System and database performance consulting
� VAX/Alpha emulation – CHARON reseller
� Migration consulting
� Specializing in minimal downtime migrations
� Oracle’s worldwide provider of CODASYL DBMS training
41
Top Related