Linux Deployment and Management under zVM at Wells Fargo...
Transcript of Linux Deployment and Management under zVM at Wells Fargo...
![Page 1: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/1.jpg)
Linux Deployment and Management under zVM at Wells Fargo Bank
Bob Bates
Wells Fargo Bank
August 15, 2013
Session Number 13976
![Page 2: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/2.jpg)
About Wells Fargo Bank
o Headquartered in San Francisco
o Acquired Wachovia Bank December 2008 (doubled) – completed
migration end of 2011
o 70 Million customers, 1/3 of US Households have relationship
o 9000 Stores (branches)
o #1 Mortgage lender (commercial and residential)
o #3 ATMs (12,355)
o #4 in US in full service brokerage & wealth management
o #5 Insurance brokerage
o #4 in assets
o #1 in market cap
o 1st to do internet banking
o 270,000 employees
2
![Page 3: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/3.jpg)
About Me
o VM user since 1977 (VM/370 R3) and Systems programmer since
1980 (VM/SP R1).
o Systems programmer for z/VM and Linux on z Series at WF
o AVP, Operating Systems Engineer
o Enterprise Hosting Services
o At WF since 2007
o 8 team members
o Install/support/engineer Linux on z and z/VM builds
o Level 2-3 for all Linux on z servers
o Sir Bob, Master Darter of Beantown
3
![Page 4: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/4.jpg)
What We Have
o Production
o 4 LPARs on 2 sites with failover between
o z196 with 52 IFLs total
o zVM 6.2
o Development/Test
o 3 LPARs on 2 boxes
oz10 2 LPARs 6 IFLs zVM 6.2 and 2 IFLs zVM 6.3
oz196 1 LPAR 6 IFLs zVM 6.3
4
![Page 5: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/5.jpg)
How It Was
Adding a server meant several tasks:
oDefine the server
oCopy the disks using Hidro or DDR
oUpdate the LINUXUP EXEC on AUTOLOG2
o Add a GRANT VSWITCH on AUTOLOG2
oUpdate the LINUX NODES file on ZTCP
oUpdate a list in LNXDOWN EXEC
oUpdate files on server’s 191
5
![Page 6: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/6.jpg)
Drawbacks
o One change could effect several files.
o Missing a step could result in problems later
o Any change to the base version needs to be tested out
before creating new servers.
o Base version was only locally available (pre-SSI) and had
to be duplicated on each system for deployment.
6
![Page 7: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/7.jpg)
First step – Save the Image
o Created an SFS pool for saving the LINUX image after it
was built.
o Accessible to all systems meaning it could be built
anywhere.
o Use CMSDDR to copy MDISK to FILE. o PIPE CMS EXEC CMSDDR DUMP addr outfn outft outfm
o Slow interface between prod and dev (1K miles apart)
required a 2nd SFS pool. One in dev, one in prod.
7
![Page 8: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/8.jpg)
SAVESYS
8
Blank lines allow for new description. User has to be enrolled in SFS to appear in list.
![Page 9: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/9.jpg)
SAVESYS
9
Removing ‘S’ prevents that disk from being saved.
![Page 10: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/10.jpg)
SAVESYS
10
Directory structure has user (os level), description, date saved.
![Page 11: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/11.jpg)
SAVESYS
11
Files are actually named to identify address and size: IMAG0103 CYL00300
![Page 12: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/12.jpg)
LXSFSWK
o Nothing to show, works in the dark of night to synchronize
the two SFS.
o Compares directory structure, user limits, and files.
o Keeps last status so it knows if something has been
deleted.
o Can run for hours, but saves tons of time when creating
images.
12
![Page 13: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/13.jpg)
LXSHARE 192 - Files Involved
o LXSHARE 192 contains several files for the build and boot processes.
o PROFILE EXEC for server calls LXSTART which prepares for boot
o LXSTART calls LXIDENT which build config file for witsec.
o LINUX_OS NAMES contains default information for each level
(SLES10, SLES11, etc)
o SUBNET NAMES contains default information for each network
o LINUX NAMES contains all the information for each server.
o Fields in LINUX_OS and SUBNET determine which fields are
displayed on BUILDSYS screens.
13
![Page 14: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/14.jpg)
BUILDSYS Creation of new server(s)
14
![Page 15: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/15.jpg)
BUILDSYS Creation of new server(s)
15
Userid is created using network ip address if desired
![Page 16: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/16.jpg)
BUILDSYS Creation of new server(s)
16
Yellow is protected and built from other values. Process causes the id creation and population of LINUX NAMES
![Page 17: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/17.jpg)
BUILDSYS Creation of new server(s)
17
Once restores complete another pfkey is revealed. PF10=Create Duplicates
![Page 18: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/18.jpg)
BUILDSYS Creation of new server(s)
18
Userids created, LINUX NAMES updated, disks created with HiDro duplicating from original userid (ZL3077)
![Page 19: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/19.jpg)
LINUX NAMES entry for BOBTEST1
19
![Page 20: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/20.jpg)
AUTOLOG2 - LINUXUP
o Servers are grouped a couple ways
o Startup: Early, Normal, Late
o Start maximum of 10 at a time
o Runsys: Name of system to start it on
o PIPE CMS NAMEFIND :runsys DVEVM :Startup Normal :nick (FILE LINUX
TYPE * | stem serv_list.
Note: If a server has no runsys or startup it will not be selected to be
autologged.
20
![Page 21: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/21.jpg)
WITSEC
File created when server is
autologged
Read at boot
WITSEC decides if anything
has changed
Makes adjustments if
needed.
21
![Page 22: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/22.jpg)
ZTCP – BLDLXNOD
o Rebuild the LINUX NODE file each time ZTCP is started
o Grab the servers for the system, the IP address, and
monitor tag = YES.
o Create the lines for each server.
node = ‘ZL3077’ network_address = ‘10.xx.zz.77’ vmid = ‘ZL3077’
o Does a copy to keep the last version just in case as well.
22
![Page 23: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/23.jpg)
LNXDOWN
Many servers in prod are still shutdown once a week and
backed up from zOS (full pack).
LNXDOWN SYSTEM PR8VM BACKUP
Must be run from PR8VM (checks you are on the right
system)
Selects all servers from system with Backup = YES
Issues SIGNAL SHUTDOWN USER server_id
23
![Page 24: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/24.jpg)
VM:Operator Check servers in the cluster
24
Green indicates server logged on. Red not logged on.
![Page 25: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/25.jpg)
VM:Operator Check servers in the cluster
25
PF5 gives information on server selected.
![Page 26: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/26.jpg)
VM:Operator Check servers in the cluster
26
PF9 toggles between Userid and Hostname.
![Page 27: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/27.jpg)
Bring up a server by request
o Servers started by request in BCP
o BRINGUP server1 server2 …
o Server server1 started on runsys
o Operators often get request from users who don’t know the
VM userid
o Looks up hostname and gets runsys and nick
o Issues AT runsys CMD XAUTOLOG nick
27
![Page 28: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/28.jpg)
LINUX NAMES
o File can get very large in a hurry.
o Adding a new field becomes tiresome.
o Manual maintenance can be dangerous.
28
![Page 29: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/29.jpg)
LNXNAMES Update/display LINUX NAMES file entries
29
![Page 30: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/30.jpg)
LNXNAMES Update/display LINUX NAMES file entries
30
Customize the view with selected data
![Page 31: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/31.jpg)
LNXNAMES Update/display LINUX NAMES file entries
31
![Page 32: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/32.jpg)
LNXNAMES Update/display LINUX NAMES file entries
32
![Page 33: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/33.jpg)
V$BLDSYS FIELDS
33
![Page 34: Linux Deployment and Management under zVM at Wells Fargo …share.confex.com/share/121/webprogram...About Me o VM user since 1977 (VM/370 R3) and Systems programmer since 1980 (VM/SP](https://reader033.fdocuments.in/reader033/viewer/2022050419/5f8f1ff24e54fe37ad4d6789/html5/thumbnails/34.jpg)
34
Questions?