How to Use Black Cat Usb

Virtual SAN Health checks Guide

S T ORAG E A ND A VA I L AB I L I T Y DO CUMENTAT I ON / 1

VMware Virtual SAN Health Check Plugin Guide Cormac Hogan Storage and Availability Business Unit VMware v 6.0.0a / May 2015

Virtual SAN Health Check Plugin Guide

S T O RAG E A ND A V A I L A B I L I T Y D O CUMENTA T I ON / 2

Contents

INTRODUCTION ...................................................................................................................................... 7 SECURITY CONSIDERATIONS ................................................................................................................................... 8 Exposed Interfaces, ports and Service ......................................................................................................... 8 Security Related Resource ................................................................................................................................ 9 Log Location ........................................................................................................................................................... 9 Log-on ID Account ............................................................................................................................................... 9 Log-on ID Account Privileges .......................................................................................................................... 9 vCenter Server Privileges ................................................................................................................................ 10 UPGRADE PROCESS ................................................................................................................................................. 11

1. INSTALLATION - VCENTER SERVER .......................................................................................... 12 1.1 INSTALLATION ON THE VCENTER APPLIANCE ........................................................................................... 12 1.1.1 Download the health check plugin RPM ....................................................................................... 12 1.1.2a Install the vsan-health RPM ............................................................................................................. 13 1.1.2b Upgrading the health check plugin RPM .................................................................................... 13 1.1.3 Run the post install script to restart the services ...................................................................... 13 1.1.4 Checking the install status via RVC ................................................................................................. 15 1.1.5 ESXi health check VIB upgrades ....................................................................................................... 16 1.2 INSTALLATION ON WINDOWS VCENTER SERVER ...................................................................................... 17 1.2.1 Download the health check plugin MSI ......................................................................................... 17 1.2.2 Run the vsan-health MSI ...................................................................................................................... 18 1.2.3 Upgrading the vsan-health MSI ........................................................................................................ 19 1.2.4 Restart the vCenter Server services ................................................................................................. 19 1.2.5 Checking the vCenter install status via RVC ................................................................................ 20

2. VIB INSTALLATION - ESXI ............................................................................................................. 21 2.1 BEFORE YOU BEGIN ......................................................................................................................................... 21 2.2 INSTALLATION METHODS .............................................................................................................................. 22 2.3 VSPHERE WEB CLIENT UI INSTALLATION METHOD ................................................................................. 23 2.3.1 Enable Health Checks via Mange -> Virtual SAN -> Health .................................................. 23 2.3.2 Monitor health checks via Monitor -> Virtual SAN -> Health .............................................. 25 2.4 RVC INSTALLATION METHOD ....................................................................................................................... 26 2.4.1 vsan.health.cluster_install ................................................................................................................... 26 2.4.2 vsan.health.cluster_status ................................................................................................................... 28 2.4.3 Changes to General View with Health check plugin ................................................................ 28 2.5 UPGRADING HEALTH CHECK ESXI VIBS .................................................................................................... 29 2.6 MANUAL VIB INSTALL ................................................................................................................................... 30 2.7 VSPHERE UPDATE MANAGER USERS WARNING ......................................................................................... 30

3. HEALTH CHECK PLUGIN UI FEATURES .................................................................................... 31 3.1 ENABLE CUSTOMER EXPERIENCE IMPROVEMENT PROGRAM ................................................................. 31 3.2 HCL DATABASE ............................................................................................................................................... 32 3.3 SUPPORT ASSISTANT ...................................................................................................................................... 32 3.4 EXTERNAL PROXY SETTINGS ......................................................................................................................... 33 4. HEALTH CHECK PLUGIN UNINSTALL ........................................................................................ 34 4.1 ESXI UNINSTALL VIA UI ................................................................................................................................. 34 4.2 ESXI UNINSTALL VIA VSAN.HEALTH.CLUSTER_UNINSTALL ..................................................................... 35



4.3 REMOVING HEALTH CHECK PLUGIN FROM VCENTER ............................................................................... 36 4.3.1 vCenter Appliance uninstall ................................................................................................................ 36 4.3.2 Windows vCenter Server uninstall ................................................................................................... 37

5. HEALTH CHECK VIA THE CLI ....................................................................................................... 38 5.1 HEALTH CHECK OVERVIEW ............................................................................................................................ 38 5.1 CHECK STATUS - VSAN.HEALTH.CLUSTER_STATUS .................................................................................... 39 5.2 DISPLAY HEALTH - VSAN.HEALTH.HEALTH_SUMMARY ............................................................................. 40 6. HEALTH CHECKS VIA THE UI ....................................................................................................... 42 6.1 HEALTH CHECK TESTS AND RESULTS VIA UI .............................................................................................. 42 6.2 WHAT DOES A FAILURE LOOK LIKE? ............................................................................................................ 43 6.3 WHAT TO DO WHEN A HEALTH CHECK FAILS? ........................................................................................... 43 6.4 CLUSTER HEALTH ESX VSAN HEALTH CHECK INSTALLATION .......................................................... 44

6.4.1 What does this check do? ..................................................................................................................... 44 6.4.2 What does it mean when it is in an error state? ........................................................................ 44 6.4.3 How does one troubleshoot and fix the error state? ................................................................ 44 6.5 CLUSTER HEALTH - VSAN HEALTH CHECKS UP-TO-DATE ..................................................................... 45 6.5.1 What does this check do? ..................................................................................................................... 45 6.5.2 What does it mean when it is in an error state? ........................................................................ 45 6.5.3 How does one troubleshoot and fix the error state? ................................................................ 45 6.6 CLUSTER HEALTH - ADVANCED VIRTUAL SAN CONFIGURATION IN SYNC ........................................... 46 6.6.1 What does this check do? ..................................................................................................................... 46 6.6.2 What does it mean when it is in an error state? ........................................................................ 46 6.6.3 How does one troubleshoot and fix the error state? ................................................................ 46 6.7 CLUSTER HEALTH - VSAN CLOMD LIVENESS .......................................................................................... 47 6.7.1 What does this check do? ..................................................................................................................... 47 6.7.2 What does it mean when it is in an error state? ........................................................................ 47 6.7.3 How does one troubleshoot and fix the error state? ................................................................ 47 6.8 VSAN HCL HEALTH VSAN HCL DB UP-TO-DATE ............................................................................... 49 6.8.1 What does this check do? ..................................................................................................................... 49 6.8.2 What does it mean when it is in an error state? ........................................................................ 49 6.8.3 How does one troubleshoot and fix the error state? ................................................................ 50 6.9 VSAN HCL HEALTH SCSI CONTROLLER ON VSAN HCL .................................................................... 51 6.9.1 What does this check do? ..................................................................................................................... 51 6.9.2 What does it mean when it is in an error state? ........................................................................ 51 6.9.3 How does one troubleshoot and fix the error state? ................................................................ 51 6.10 VSAN HCL HEALTH CONTROLLER RELEASE SUPPORT ..................................................................... 52 6.10.1 What does this check do? .................................................................................................................. 52 6.10.2 What does it mean when it is in an error state? ..................................................................... 52 6.10.3 How does one troubleshoot and fix the error state? ............................................................. 52 6.11 VSAN HCL HEALTH CONTROLLER DRIVER ......................................................................................... 53 6.11.1 What does this check do? .................................................................................................................. 53 6.11.2 What does it mean when it is in an error state? ..................................................................... 53 6.11.3 How does one troubleshoot and fix the error state? ............................................................. 53 6.12 HCL HEALTH HOST ISSUES RETRIEVING HARDWARE INFO ............................................................... 54 6.12.1 What does this check do? .................................................................................................................. 54 6.12.2 What does it mean when it is in an error state? ..................................................................... 54 6.12.3 How does one troubleshoot and fix the error state? ............................................................. 54 6.13 NETWORK HEALTH - HOSTS DISCONNECTED FROM VCENTER ............................................................. 55 6.13.1 What does this check do? .................................................................................................................. 55



6.13.2 What does it mean when it is in an error state? ..................................................................... 55 6.13.3 How does one troubleshoot and fix the error state? ............................................................. 55 6.14 NETWORK HEALTH HOSTS WITH CONNECTIVITY ISSUES .................................................................. 56 6.14.1 What does this check do? .................................................................................................................. 56 6.14.2 What does it mean when it is in an error state? ..................................................................... 56 6.14.3 How does one troubleshoot and fix the error state? ............................................................. 56 6.15 NETWORK HEALTH VSAN CLUSTER PARTITION ................................................................................. 57 6.15.1 What does this check do? .................................................................................................................. 57 6.15.2 What does it mean when it is in an error state? ..................................................................... 57 6.15.3 How does one troubleshoot and fix the error state? ............................................................. 58 6.16 NETWORK HEALTH UNEXPECTED VSAN CLUSTER MEMBERS ......................................................... 59 6.16.1 What does this check do? .................................................................................................................. 59 6.16.2 What does it mean when it is in an error state? ..................................................................... 59 6.16.3 How does one troubleshoot and fix the error state? ............................................................. 60 6.17 NETWORK HEALTH HOSTS WITH VSAN DISABLED ............................................................................ 61 6.17.1 What does this check do? .................................................................................................................. 61 6.17.2 What does it mean when it is in an error state? ..................................................................... 61 6.17.3 How does one troubleshoot and fix the error state? ............................................................. 61 6.18 NETWORK HEALTH ALL HOSTS HAVE A VSAN VMKNIC CONFIGURED ............................................ 63 6.18.1 What does this check do? .................................................................................................................. 63 6.18.2 What does it mean when it is in an error state? ..................................................................... 63 6.18.3 How does one troubleshoot and fix the error state? ............................................................. 63 6.19 NETWORK HEALTH ALL HOSTS HAVE MATCHING SUBNETS .............................................................. 65 6.19.1 What does this check do? .................................................................................................................. 65 6.19.2 What does it mean when it is in an error state? ..................................................................... 65 6.19.3 How does one troubleshoot and fix the error state? ............................................................. 65 6.20 NETWORK HEALTH ALL HOSTS HAVE MATCHING MULTICAST SETTINGS ....................................... 67 6.20.1 What does this check do? .................................................................................................................. 67 6.20.2 What does it mean when it is in an error state? ..................................................................... 67 6.20.3 How does one troubleshoot and fix the error state? ............................................................. 67 6.21 NETWORK HEALTH BASIC (UNICAST) CONNECTIVITY CHECK (NORMAL PING) ............................ 69 6.21.1 What does this check do? .................................................................................................................. 69 6.21.2 What does it mean when it is in an error state? ..................................................................... 69 6.21.3 How does one troubleshoot and fix the error state? ............................................................. 69 6.22 NETWORK HEALTH MTU CHECK (PING WITH LARGE PACKET SIZE) ............................................... 70 6.22.1 What does this check do? .................................................................................................................. 70 6.22.2 What does it mean when it is in an error state? ..................................................................... 70 6.22.3 How does one troubleshoot and fix the error state? ............................................................. 70 6.23 NETWORK HEALTH MULTICAST ASSESSMENT BASED ON OTHER CHECKS ....................................... 71 6.23.1 What does this check do? .................................................................................................................. 71 6.23.2 What does it mean when it is in an error state? ..................................................................... 71 6.23.3 How does one troubleshoot and fix the error state? ............................................................. 71 6.24 NETWORK HEALTH ACTIVE MULTICAST CONNECTIVITY CHECK ...................................................... 72 6.24.1 What does this check do? .................................................................................................................. 72 6.24.2 What does it mean when it is in an error state? ..................................................................... 72 6.24.3 How does one troubleshoot and fix the error state? ............................................................. 72 6.25 DATA HEALTH VIRTUAL SAN OBJECT HEALTH .................................................................................. 74 6.25.1 What does this check do? .................................................................................................................. 74 6.25.2 What does it mean when an object is not in a healthy state? ........................................... 74 6.25.3 How does one troubleshoot and fix the error state? ............................................................. 76



6.26 LIMITS HEALTH CURRENT CLUSTER SITUATION ................................................................................. 77 6.26.1 What does this check do? .................................................................................................................. 77 6.26.2 What does it mean when it is in an error state? ..................................................................... 77 6.26.3 How does one troubleshoot and fix the error state? ............................................................. 77 6.27 LIMITS HEALTH AFTER 1 ADDITIONAL HOST FAILURE ...................................................................... 79 6.27.1 What does this check do? .................................................................................................................. 79 6.27.2 What does it mean when it is in an error state? ..................................................................... 79 6.27.3 How does one troubleshoot and fix the error state? ............................................................. 79 6.28 PHYSICAL DISK HEALTH OVERALL DISKS HEALTH ............................................................................. 80 6.28.1 What does this check do? .................................................................................................................. 80 6.28.2 What does it mean when it is in an error state? ..................................................................... 80 6.29 PHYSICAL DISK HEALTH METADATA HEALTH ..................................................................................... 81 6.29.1 What does this check do? .................................................................................................................. 81 6.29.2 What does it mean when it is in an error state? ..................................................................... 81 6.29.3 How does one troubleshoot and fix the error state? ............................................................. 81 6.30 PHYSICAL DISK HEALTH DISK CAPACITY .............................................................................................. 82 6.30.1 What does this check do? .................................................................................................................. 82 6.30.2 What does it mean when it is in an error state? ..................................................................... 82 6.30.3 How does one troubleshoot and fix the error state? ............................................................. 82 6.31 PHYSICAL DISK HEALTH SOFTWARE STATE HEALTH ......................................................................... 84 6.31.1 What does this check do? .................................................................................................................. 84 6.31.2 What does it mean when it is in an error state? ..................................................................... 84 6.31.3 How does one troubleshoot and fix the error state? ............................................................. 84 6.32 PHYSICAL DISK HEALTH CONGESTION .................................................................................................. 85 6.32.1 What does this check do? .................................................................................................................. 85 6.32.2 What does it mean when it is in an error state? ..................................................................... 85 6.32.3 How does one troubleshoot and fix the error state? ............................................................. 85 6.33 PHYSICAL DISK HEALTH COMPONENT METADATA HEALTH .............................................................. 86 6.33.1 What does this check do? .................................................................................................................. 86 6.33.2 What does it mean when it is in an error state? ..................................................................... 86 6.33.3 How does one troubleshoot and fix the error state? ............................................................. 86 6.34 PHYSICAL DISK HEALTH MEMORY POOLS (HEAPS) ............................................................................. 87 6.34.1 What does this check do? .................................................................................................................. 87 6.34.2 What does it mean when it is in an error state? ..................................................................... 87 6.34.3 How does one troubleshoot and fix the error state? ............................................................. 87 6.35 PHYSICAL DISK HEALTH MEMORY POOLS SLABS ................................................................................. 88 6.35.1 What does this check do? .................................................................................................................. 88 6.35.2 What does it mean when it is in an error state? ..................................................................... 88 6.35.3 How does one troubleshoot and fix the error state? ............................................................. 88 6.36 PHYSICAL DISK HEALTH PHYSICAL DISK HEALTH RETRIEVAL ISSUES ........................................... 89 6.36.1 What does this check do? .................................................................................................................. 89 6.36.2 What does it mean when it is in an error state? ..................................................................... 89 6.36.3 How does one troubleshoot and fix the error state? ............................................................. 89

7. PROACTIVE HEALTH CHECKS ..................................................................................................... 90 7.1 VM CREATION TESTS ...................................................................................................................................... 90 7.1.1 Running a create vm test via the UI ................................................................................................ 90 7.1.2 Running a create vm test via the CLI .............................................................................................. 92 7.2 MULTICAST PERFORMANCE TEST ................................................................................................................. 93 7.2.1 Running a multicast speed test via the UI .................................................................................... 94



7.2.2 Running a multicast speed test via the CLI .................................................................................. 95 7.3 STORAGE PERFORMANCE TEST ..................................................................................................................... 96 7.3.1 A note about storage performance tests ....................................................................................... 96 7.3.2 Running storage performance test from the UI ......................................................................... 98 7.3.3 What constitutes a pass or failure? .............................................................................................. 100 7.3.4 Using Storage Performance Tests as Burn-in Tests .............................................................. 100 7.3.5 Running a storage performance test from the CLI ................................................................ 101 7.3.6 Working Example I 3 steps ........................................................................................................... 102 7.3.7 Working Example II 1 step ........................................................................................................... 106 7.4 HOST DEBUG MULTICAST ........................................................................................................................... 107 7.4.1 Running a multicast debug test via the CLI .............................................................................. 107

8. ADDITIONAL INFORMATION .................................................................................................... 108 8.1 DOCUMENTATION ......................................................................................................................................... 108 8.2 VIRTUAL SAN RESOURCES ......................................................................................................................... 108 8.3 SUPPORT ISSUES ........................................................................................................................................... 108 8.4 WHERE DOES ONE LOCATE THE LATEST HEALTH CHECK PLUGIN SOFTWARE? ............................... 108 8.5 WHERE DOES ONE LOCATE THE LATEST HCL FILE? .............................................................................. 108



Introduction VMwares Virtual SAN is designed to be simple: simple to configure, and simple to operate. This simplicity masks a sophisticated and powerful storage product. This simplicity hides a lot of the complexities found in most modern day storage systems. Under the covers, Virtual SAN also has its complexities. While VMware does provide extensive documentation in the form of a Virtual SAN Administrators Guide and Virtual SAN Troubleshooting Reference Manual, there is a desire to continue with the simple approach and highlight, through Health check plugin, the component that may be at fault. Virtual SAN Health check plugin checks all aspects of a Virtual SAN configuration. It implements a number of checks on hardware compatibility, networking configuration and operations, advanced Virtual SAN configuration options, storage device health as well as virtual machine object health. The health check will provide two main benefits to administrators of Virtual SAN environments: 1. It will give administrators peace of mind that their Virtual SAN deployment is fully supported, functional and operational 2. It will provide immediate indications to a root cause in the event of a failure, leading to speedier resolution times The purpose of this document is to fully illustrate Virtual SAN Health check plugin. The document will cover how to install the feature, and how to use the feature. It will also provide some context related to the issue highlighted and how t go about problem solving on Virtual SAN. It is recommended that the Health check plugin be utilized to do initial triage of any Virtual SAN problems. For those who wish to learn more about diagnosis and troubleshooting on Virtual SAN, the Virtual SAN Troubleshooting Reference Manual is a good starting point. There is a link at the end of this guide. Health check plugin provides users with the ability to join VMwares Customer Experience Improvement Program (CEIP). This is optional, and can be enabled or disabled at any time through the Health check plugin UI. One final note is that a new feature called the Virtual SAN Support Assistant is also included. This allows administrators to upload VSAN log bundles directly to their support request (SR) opened with VMware Global Support Services (GSS). Note: Presently, the Virtual SAN Health Services Plugin is only supported with Virtual SAN version 6.0. It is not supported with Virtual SAN version 5.5 at this time.



Security Considerations This section will describe the detailed security related information for Virtual SAN health check plugin, including: 1. A list of all external interfaces, ports, and services and which ones need to be open or enabled for proper operation of the product. 2. All resources that need to be protected such as security-relevant configuration files and passwords and the recommended access controls for secure operation. 3. Location of log files and description of how to interpret security related log messages 4. Log-on ID of accounts created during system install/bootstrap and instructions on how to change defaults. 5. Privileges assigned to service users. 6. Information on how customers can obtain and apply the latest security update/patch. Exposed Interfaces, ports and Service Virtual SAN health check plugin builds a standalone python web service to expose a new managed object "vsan-cluster-health-system" and user can get through the URL https://VC_IP/vsanHealth to invoke the Virtual SAN health check plugin APIs. The python web server will open the local port 8005 on the vCenter Server and register it in the reverse proxy so that all of http request under https://VC_IP/vsanHealth will route to the Virtual SAN health check plugin. Since Virtual SAN health service only opens local ports without external access, there should be no security concern here. From an ESXi perspective, when the network multicast performance test is triggered from the Virtual SAN Health Check UI or via the relevant Ruby vSphere Console (RVC) command, port 5001 is temporarily opened on each ESXi host in the Virtual SAN cluster. This port is closed when the test completes. No ports are opened on vCenter Server side for this test.



Security Related Resource VIRTUAL SAN Health check plugin has a configuration file on the vCenter Appliance in/usr/lib/vmware-vpx/vsan-health/config.conf that may include the configuration item "proxy password" providing by user, which is security sensitive. On the vCenter Server for Windows, the file is located in %VMWARE_CIS_HOME%\vsan-health\config.conf. Currently we use a file system based protection mechanism, so that the Virtual SAN health user only has the privilege to read and write to this file, preventing other non-privileged users from viewing the password. Log Location The default log location for Virtual SAN health check plugin is /var/log/vmware/vsan-health. And user can change it by modifying the configuration item "logdir" in the configuration file under /usr/lib/vmware-vpx/vsan-health. On the vCenter Server for Windows, the file is located in %VMWARE_LOG_DIR%\vsan-health. No security related information is logged in the log file. Log-on ID Account On the vCenter Appliance, Virtual SAN Health check plugin creates a local user (vsan-health) under Users (100) group. This happens by default when first booting Virtual SAN health check plugin. It cannot be changed. There is no new user created on the vCenter Server for Windows. Log-on ID Account Privileges The vsan-health account will be added into CIS group so that it has the privilege to read/write/execute the java and relevant vCenter Server Appliance related files (e.g. cloudvm-ram-size.log). Also it has the read/execute privilege for its installation root directory, and read/write privilege for the Virtual SAN Health check log and configuration directory.


S T ORAG E A ND A VA I L AB I L I T Y DO CUMENTAT I ON / 1 0

vCenter Server Privileges This is the summary of the privilege that will be required by the vCenter server user who wishes to run the Virtual SAN health checks. 1. Cluster System.View/Read The Cluster System.View/Read privileges are needed to be able to carry out the following health checks.

All Query* API which include all health checks and proactive tests except for creating VM test Enable/Disable VSAN health, Enable/Disable Phone-Home, proxy setting.

o The UI will apply the cluster-edit privilege check again to disable such operation if user hasn't proper privileges). 2. Cluster System.View/Read + Global.Setting The Cluster System.View/Read and the Global.Setting privileges are needed to be able to carry out the following health checks.

SendVsanTelemetry UploadHclDB From File Upload HCL db From Web

3. Cluster System.View/Read + Global.Diagnostics These Cluster System.View/Read and the Global .Diagnostics privileges are needed to be able to carry out the following health check.

AttachToSR/PR 4. Cluster System.View/Read + Virtual Machine.Create/Delete The Cluster System.View/Read and the Virtual Machine.Create/Delete privileges are needed to be able to carry out the following health checks.

Create VM Test



Upgrade Process The upgrade process for Virtual SAN health service on the vCenter Server appliance involves two distinct steps. The first step is the downloading of a new Virtual SAN Health service RPM, upgrading the RPM, and then restarting the Virtual SAN health service. The next step is for the user to open the Virtual SAN health main page through Cluster > Manage > Settings > Virtual VSAN > Health, and click the upgrade button. For the Windows version of the vCenter Server, there are also two distinct steps for upgrading Virtual SAN health check plugin. The first step is to download the new Virtual SAN health check MSI installer, and run the installer. The installer will auto detect older versions of the health check and do the upgrade. After the installer finishes the upgrade, the VMware vCenter Service (VC) must be manually restarted in the Windows services manager. The second step is the same as the vCenter appliance, namely that the user needs to open the Virtual SAN health service main page through Cluster > Manage > Settings > Virtual VSAN > Health, and click the upgrade button to upgrade the hosts in the Virtual SAN cluster. The upgrade process is covered in greater detail later on in this guide.



1. Installation - vCenter Server This chapter will cover the first installation step, which is deploying the health check plugin on the vCenter server. The health service plugin is supported on both the vCenter appliance and the Windows version of vCenter server. First, we shall look at the steps involved in installing the plugin on the appliance, and then we shall look at the steps involved in deploying it on Windows. 1.1 Installation on the vCenter Appliance There are a number of different steps in deploying the Virtual SAN Health check plugin. In this case, the steps to deploy the plugin on a Virtual SAN deployment that is managed by the vCenter server (Linux) appliance are examined. The steps can be broken into four distinct parts: 1. Download the health check plugin RPM to the vCenter server appliance 2. Deploy the health check plugin RPM 3. Run the post install script and restart the vCenter server services 4. Check the status of the install Below are the detailed steps for deploying the health check plugin on the vCenter server appliance. The references will be replaced with the actual released build number of the software. The RPM is available from the following location: https://my.vmware.com/group/vmware/get-download?downloadGroup=VSANHEALTH600 1.1.1 Download the health check plugin RPM The Health check plugin RPM for vCenter server appliance can be retrieved in a number of different ways. Since the appliance provides a utility called wget, this can be used to retrieve the RPM directly if your vCenter server appliance has Internet access. # wget http:///VMware-vsan-health-6.0.0-.x86_64.rpm --2015-01-15 15:58:25-- http:///VMware-vsan-health-6.0.0-.x86_64.rpm Connecting to 10.116.224.13:80... connected. HTTP request sent, awaiting response... 200 OK Length: 9725916 (9.3M) [text/plain] Saving to: `VMware-vsan-health-6.0.0-.x86_64.rpm' 100%[======================================>] 9,725,916 3.45M/s in 2.7s 2015-01-15 15:58:29 (3.45 MB/s) - `VMware-vsan-health-6.0.0-.x86_64.rpm' saved [9725916/9725916] # ls .bash_history .odbc.ini .viminfo .exrc .pgpass VMware-vsan-health-6.0.0-.x86_64.rpm .gnupg .pyhistory inst-sys .kbd .rvc vimtop .lesshst .rvc-history #



Determine if this is a new install or an upgrade. If it is an install, go to step 2a. If it is an upgrade, go to step 2b. 1.1.2a Install the vsan-health RPM If you are installing the health check plugin for the first time, the command rpm Uvh can be used to achieve this: # rpm -Uvh VMware-vsan-health-6.0.0-.x86_64.rpm Preparing... ########################################### [100%] 1:VMware-vsan-health ########################################### [100%]

1.1.2b Upgrading the health check plugin RPM If you are upgrading an existing health check plugin RPM, it is simply a matter of running the same rpm command as step 2a but with the option -replacepkgs: # rpm -Uvh --replacepkgs VMware-vsan-health-6.0.0-.x86_64.rpm warning: package VMware-vsan-health = 6.0.0- was already added, replacing with VMware-vsan-health >= 6.0.0-2439785 Preparing... ########################################### [100%] 1:VMware-vsan-health ########################################### [100%] # 1.1.3 Run the post install script to restart the services Once the health check plugin rpm has been installed on the vCenter server appliance, the vpxd and vsphere-client services need to be restarted. A script is provided to do this step. # /usr/lib/vmware-vpx/vsan-health/health-rpm-post-install.sh Services 'vpxd' and 'vsphere-client' need to be restarted. This will disrupt all ongoing sessions. Do you want to continue? [Yes/No]? Yes /usr/lib/vmware-vpx/workflow/bin 2015-04-20T08:55:41.371Z Getting value for install-parameter: workflow.int.service-port 2015-04-20T08:55:41.387Z Getting value for install-parameter: workflow.int.jmx-port 2015-04-20T08:55:41.400Z Getting value for install-parameter: vpxd.int.sdk-port 2015-04-20T08:55:41.408Z Getting value for install-parameter: vpxd.int.sdk-tunnel-port 2015-04-20T08:55:41.417Z Getting value for install-parameter: rhttpproxy.ext.port1 2015-04-20T08:55:41.426Z Getting value for install-parameter: rhttpproxy.ext.port2 {'vpxd_sdk_tunnel_port': '8089', 'rhttpproxy_https_port': '443', 'rhttpproxy_http_port': '80', 'workflow_service_port': '8088', 'vpxd_sdk_port': '8085', 'PASSWORD': '', 'workflow_jmx_port': '19999'} 2015-04-20T08:55:41.434Z Getting value for install-parameter: syslog.ext.port 2015-04-20T08:55:41.443Z Getting value for install-parameter: vc.home.path 2015-04-20T08:55:41.453Z Getting value for install-parameter: vc.conf.path 2015-04-20T08:55:41.454Z VSAN Health service firstboot started 2015-04-20T08:55:41.463Z User %s already exists, skipping creation. 2015-04-20T08:55:41.475Z Getting value for install-parameter: rhttpproxy.cert



2015-04-20T08:55:41.486Z Running command: ['/usr/bin/openssl', 'x509', '-noout', '-in', '/etc/vmware-rhttpproxy/ssl/rui.crt', '-fingerprint'] 2015-04-20T08:55:41.741Z Done running command None 2015-04-20T08:55:42.594Z Waiting for vpxd to initialize Waiting for vim service at 443 2015-04-20T08:55:42.620Z Done waiting for vpxd to initialize 2015-04-20T08:55:42.620Z Installing Virtual SAN Health Service 2015-04-20T08:55:42.620Z Running command: ['/sbin/chkconfig', 'vmware-vsan-health', 'on'] 2015-04-20T08:55:42.759Z Done running command 2015-04-20T08:55:42.759Z Running command: ['usermod', '-A', 'cis', 'vsan-health'] 2015-04-20T08:55:42.826Z Done running command 2015-04-20T08:55:42.826Z Running command: ['chgrp', 'cis', '/usr/java/jre-vmware/bin'] 2015-04-20T08:55:42.836Z Done running command 2015-04-20T08:55:42.836Z Running command: ['chmod', 'g-rwx', '/usr/java/jre-vmware/bin'] 2015-04-20T08:55:42.840Z Done running command 2015-04-20T08:55:42.841Z Running command: ['chmod', 'g+rwx', '/usr/java/jre-vmware/bin'] 2015-04-20T08:55:42.844Z Done running command 2015-04-20T08:55:42.844Z Running command: ['chgrp', 'cis', '/var/log/vmware/cloudvm/cloudvm-ram-size.log'] 2015-04-20T08:55:42.848Z Done running command 2015-04-20T08:55:42.848Z Running command: ['chmod', 'g-rwx', '/var/log/vmware/cloudvm/cloudvm-ram-size.log'] 2015-04-20T08:55:42.851Z Done running command 2015-04-20T08:55:42.851Z Running command: ['chmod', 'g+rw', '/var/log/vmware/cloudvm/cloudvm-ram-size.log'] 2015-04-20T08:55:42.854Z Done running command 2015-04-20T08:55:42.854Z Running command: ['chown', '-R', 'vsan-health', '/usr/lib/vmware-vpx/vsan-health/'] 2015-04-20T08:55:42.861Z Done running command 2015-04-20T08:55:42.861Z Running command: ['chmod', '-R', 'u-rwx', '/usr/lib/vmware-vpx/vsan-health/'] 2015-04-20T08:55:42.865Z Done running command 2015-04-20T08:55:42.865Z Running command: ['chmod', '-R', 'u+rx', '/usr/lib/vmware-vpx/vsan-health/'] 2015-04-20T08:55:42.869Z Done running command 2015-04-20T08:55:42.869Z Running command: ['chown', '-R', 'vsan-health', '/var/log/vmware/vsan-health/'] 2015-04-20T08:55:42.883Z Done running command 2015-04-20T08:55:42.883Z Running command: ['chmod', '-R', 'u-rwx', '/var/log/vmware/vsan-health/'] 2015-04-20T08:55:42.887Z Done running command 2015-04-20T08:55:42.887Z Running command: ['chmod', '-R', 'u+rw', '/var/log/vmware/vsan-health/'] 2015-04-20T08:55:42.890Z Done running command 2015-04-20T08:55:42.890Z Running command: ['chown', 'vsan-health', '/var/log/vmware/vsan-health/'] 2015-04-20T08:55:42.894Z Done running command 2015-04-20T08:55:42.894Z Running command: ['chmod', 'u-rwx', '/var/log/vmware/vsan-health/'] 2015-04-20T08:55:42.897Z Done running command 2015-04-20T08:55:42.898Z Running command: ['chmod', 'u+rwx', '/var/log/vmware/vsan-health/'] 2015-04-20T08:55:42.902Z Done running command 2015-04-20T08:55:42.956Z Getting value for install-parameter: vc.home.path 2015-04-20T08:55:42.956Z Stopping vsan-health 2015-04-20T08:55:42.956Z Running command: ['/sbin/service', 'vmware-vsan-health', 'stop'] 2015-04-20T08:55:46.069Z Done running command 2015-04-20T08:55:46.069Z Successfully stopped service vmware-vsan-health 2015-04-20T08:55:46.069Z Starting vsan-health 2015-04-20T08:55:46.070Z Running command: ['/sbin/chkconfig', '--force', 'vmware-vsan-health', 'on'] 2015-04-20T08:55:46.117Z Done running command 2015-04-20T08:55:46.118Z Running command: ['/sbin/service', 'vmware-vsan-health', 'start'] 2015-04-20T08:55:46.128Z Done running command 2015-04-20T08:55:46.128Z Successfully started service vmware-vsan-health



2015-04-20T08:55:46.128Z VSAN Health firstboot successfully finished vmware-vpxd: Stopping vpxd by administrative request. process id was 9254 success vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd Waiting for the embedded database to start up: success Executing pre-startup scripts... vmware-vpxd: Starting vpxd by administrative request. success vmware-vpxd: Waiting for vpxd to start listening for requests on 8089 Waiting for vpxd to initialize: .success vmware-vpxd: vpxd has initialized. Stopping VMware vSphere Web Client... Stopped VMware vSphere Web Client. Last login: Mon Apr 20 08:56:02 UTC 2015 on pts/0 Starting VMware vSphere Web Client... Waiting for VMware vSphere Web Client....... running: PID:16251 #

1.1.4 Checking the install status via RVC Once the services have been restarted, a quick way to check that the health check RPM has been successfully installed on the vCenter server is to use the following Ruby vSphere Console (RVC) commands against the cluster. For details on how to access the RVC, please refer to the Troubleshooting Reference Manual or the RVC Command Reference Guide for Virtual SAN. > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: not installed (incomplete, see issues) Issues: Unable to access agent offline bundle at https://localhost/vsanHealth/offline-bundle-6.0.0.zip Unable to access agent offline bundle at https://localhost/vsanHealth/offline-bundle-6.0.0.zip Unable to access agent offline bundle at https://localhost/vsanHealth/offline-bundle-6.0.0.zip Per-Host details: Host 'cs-ie-h01.ie.local': Status: red Issues: Unable to access agent offline bundle at https://localhost/vsanHealth/offline-bundle-6.0.0.zip Host 'cs-ie-h02.ie.local': Status: red Issues: Unable to access agent offline bundle at https://localhost/vsanHealth/offline-bundle-6.0.0.zip Host 'cs-ie-h03.ie.local': Status: red Issues: Unable to access agent offline bundle at https://localhost/vsanHealth/offline-bundle-6.0.0.zip Host 'cs-ie-h04.ie.local': Status: green Host 'cs-ie-h02.ie.local' doesn't have health system installed Host 'cs-ie-h03.ie.local' doesn't have health system installed Host 'cs-ie-h04.ie.local' doesn't have health system installed Host 'cs-ie-h01.ie.local' doesn't have health system installed vCenter Server has health system version '6.0.0' installed >



As you can see the health check has been installed on the vCenter server: vCenter Server has health system version '6.0.0' installed However it has not yet been installed on the ESXi hosts at this time. 1.1.5 ESXi health check VIB upgrades You might also come across a situation where the RPM has been upgraded on vCenter Appliance, but has not been updated on the ESXi hosts. The output would look similar to the following: > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: installed (incomplete, see issues) Per-Host details: Host 'esx-04a.corp.local': Status: green Host 'esx-01a.corp.local': Status: green Host 'esx-03a.corp.local': Status: green Host 'esx-02a.corp.local': Status: green Host 'esx-01a.corp.local' has health system version '0.3' installed Host 'esx-03a.corp.local' has health system version '0.3' installed Host 'esx-04a.corp.local' has health system version '0.3' installed Host 'esx-02a.corp.local' has health system version '0.3' installed vCenter Server has health system version '0.4' installed In this case, an upgrade is required. Note that the version numbers shown here are from a pre-released version of the health check. Installation and upgrading of ESXi hosts for the Virtual SAN health check plugin is covered in the section 3.



1.2 Installation on Windows vCenter server As we saw in the vCenter Server appliance section, there are a number of different steps in deploying the Virtual SAN Health check plugin. In this section, the steps to deploy the plugin on a Virtual SAN deployment that is managed by a Microsoft Windows based vCenter server are examined. The steps can be broken into four distinct parts: 1. Download the health check plugin MSI to the vCenter server Windows Guest OS 2. Run the health check plugin MSI 3. Manually restart the vCenter server services 4. Check the status of the install Below are the detailed steps for deploying the health check plugin on the Windows version of vCenter server. Any build number references will be replaced with the actual released build number of the software when it becomes generally available. The MSI is available from the following location: https://my.vmware.com/group/vmware/get-download?downloadGroup=VSANHEALTH600 When installing the MSI on a Windows vCenter server, the following considerations regarding the user ID must be taken into account: The user ID needs to have vCenter SSO rights The user ID must have local Windows administrator privileges

1.2.1 Download the health check plugin MSI Once the MSI file has been downloaded, it must be run on the vCenter server. Next, determine if this is a new install or an upgrade. If it is an install, go to step 2a. If it is an upgrade, go to step 2b.



1.2.2 Run the vsan-health MSI Install the downloaded MSI file by double clicking on it. Since this is a Windows Service, make sure that the user id has both vCenter SSO administrative rights and local Windows administrator privileges when you install it. You may need to open a Windows command prompt with Run as Administrator, and then run the MSI from the command prompt. The installation requires very little user interaction in this initial release of the installer. There are only two screens in the install process. One involves accepting the EULA by checking a box, as shown here:

Then click on the Install button. Once the install completes, you need to click the Finish button:



Once the install is finished, administrators can check the installation status through the Add/Remove Programs view:

The Virtual SAN health check plugin deploys by default to C:\Program Files\VMware\vCenter Server\vsan-health. Here you will find various configuration files and python (.py) scripts related to the health check plugin:

1.2.3 Upgrading the vsan-health MSI Follow the same steps as the install if you are upgrading the health check plugin. The installer will detect an already existing health check plugin and upgrade it. 1.2.4 Restart the vCenter Server services Once the install completes, the vCenter Server services, such as vpxd and vsphere-client, must be restarted. Note that the MSI for Windows does not restart the vCenter services, unlike the RPM version on the vCenter appliance.



From "Start" menu, select "Computer Management > Service and Applications-> Services". In the "Services" windows, locate a service called "VMware VirtualCenter Server", and then right click on "Restart". That completes the installation of the health check plugin on Windows vCenter server. 1.2.5 Checking the vCenter install status via RVC Once the services have been restarted, a quick way to check that the health check MSI has been successfully installed on the vCenter server is to use the Ruby vSphere Console (RVC) command, vsan.health.cluster_status. For details on how to access the RVC, please refer to the Troubleshooting Reference Manual or the RVC Command Reference Guide for Virtual SAN. There are links to these documents at the end of this guide. > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: installed (OK) Host 'cs-ie-h04.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h01.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h03.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h02.ie.local' has health system version '6.0.0' installed vCenter Server has health system version '6.0.0' installed >



2. VIB Installation - ESXi A VIB, or VMware Installation Bundle, is a VMware software patch. Each ESXi host in the Virtual SAN cluster will have a VIB installed as part of the health check plugin installation process. 2.1 Before you begin Note: The ESXi hosts in the Virtual SAN cluster need to be booted from disk. Stateless ESXi is currently not supported with Virtual SAN. If the ESXi hosts are booted from the network (e.g. stateless ESXi), the installation of VIBs will fail. Ensure that there is no rebuild taking place in the cluster before deploying the health check VIBs on the ESXi hosts in the Virtual SAN cluster. If there is, this can elongate the time to install the VIBs. Use the Ruby vSphere Console (RVC) command vsan.resync_dashboard to verify. It should report 0% activity. If there is rebuild activity, wait for it to complete before installing the health check VIBs. For details on how to access RVC, please refer to the Virtual SAN 6.0 Troubleshooting Reference Manual or RVC Reference Guide, links to which are found at the end of this guide. There is also a requirement for DRS to be enabled in fully automated mode on the cluster. The installation will not proceed unless this requirement is met.

Note: Various errors will be observed in tasks and events during this installation process. This is due to the fact that the hosts require rebooting, and the other in the hosts in the Virtual SAN cluster will complain when they cannot communicate to the host being rebooted. This is normal behavior, and is not a cause for concern. The hosts will automatically rejoin the cluster once they are successfully rebooted.



2.2 Installation Methods There are four installation options for the health VIB: 1. Enable health check plugin via vSphere web client 2. RVC, the Ruby vSphere Console 3. VUM, the VMware Update Manager. Use of VUM is outside the scope of this document and users are directed to the official vSphere documentation on how to use VUM to install VIBs. 4. Manual install at the ESXi command line. This is done using esxcli software vib commands. Use of these commands is outside the scope of this document and users are directed to the official vSphere documentation on how to use esxcli commands. VMware recommends option 1 for ease of install, but the steps to install via option 2 are also covered here.

Warning: Considering that this installation is doing a rolling upgrade, administrators should check that there is no component rebuild going on in the Virtual SAN cluster before starting this installation process on the ESXi hosts. This can be done via the UI or via the RVC command line To check if components are in a resync or rebuild state, select the cluster object, then Monitor > Virtual SAN > Resyncing components. It should be empty, as shown below:



An alternative way to check resync activity is via the RVC vsan.resync_dashboard : > vsan.resync_dashboard 2015-04-20 09:05:20 +0000: Querying all VMs on VSAN ... 2015-04-20 09:05:20 +0000: Querying all objects in the system from cs-ie-h03.ie.local ... 2015-04-20 09:05:20 +0000: Got all the info, computing table ... +-----------+-----------------+---------------+ | VM/Object | Syncing objects | Bytes to sync | +-----------+-----------------+---------------+ +-----------+-----------------+---------------+ | Total | 0 | 0.00 GB | +-----------+-----------------+---------------+ If there is rebuilding or resync activity taking place, please wait for that activity to complete before installing the health check plugin VIBs. 2.3 vSphere Web Client UI installation method The necessary Health check plugin VIBs can be installed on the ESXi hosts via the vSphere web client UI. This can be done from two places in the UI. 2.3.1 Enable Health Checks via Mange -> Virtual SAN -> Health This is the first location where the health checks can be installed. Once the cluster object is selected in the inventory, navigate to Manage -> Virtual SAN -> Health, Health check status is shown as Disabled. Click on the Enable button to enable the health check plugin. This will begin the rolling install of the health check VIB on all of the ESXi hosts in the Virtual SAN cluster.

A pop-up window is displayed to ensure you are certain you wish to enable health check plugin. As part of the health check deployment, an option to participate in the



Customer Experience Improvement Program (CEIP) is also displayed. Details about the information collected are displayed in the window. If you do not wish to participate in the Customer Experience Improvement Program at this time, simply uncheck the check box. If you wish to participate at a later stage, this is possible via the Health Check Manage view shown above.

As the dialog states, the Customer Experience Improvement Program allows VMware to capture information about how you use your vSphere environment. The data that this process collects is detailed in following location. http://www.vmware.com/info?id=1379 As the health check installation proceeds, the status changes to Enabling and once the install is complete, it changes to Enabled. This can take some time, as each ESXi host has to be placed into maintenance mode, have the health check VIB installed, and then the ESXi host has to be rebooted. The time taken to install the VIB rises exponentially based on the number of hosts in the cluster. The progress may be monitored via the Tasks & Events view of the cluster.



2.3.2 Monitor health checks via Monitor -> Virtual SAN -> Health This is the second location where the health checks can be installed. Again, the cluster object is selected in the inventory, under Monitor > Virtual SAN > Health. The service is shown as disabled. Click on Enable now to start the roll out of the health check VIBs on the ESXi hosts in the Virtual SAN cluster.

The progress of the VIB installation (maintenance mode, reboots) may once again be monitored via the Tasks & Events view of the cluster. When complete, the Health check should show Enabled in the Cluster > Manage > Virtual SAN > Health view. Navigating to Cluster > Monitor > Virtual SAN, Health should display a list of health checks and associated test results:

Note that if you have upgraded from the previous version of health check, you may need to click on the Retest button for the results to display correctly. The Virtual SAN Health check is now successfully deployed.



2.4 RVC Installation Method To install via RVC, DRS need to be enabled in fully automated mode on the cluster. Customers will have to use manual methods to install the VIB if they don't want a fully automated DRS mode on their Virtual SAN cluster. Once again, check that there is no rebuild or resync taking place. If there is, wait for it to complete. The installation is via a rolling method. It will first place a host into maintenance mode, using the ensure accessibility option. It will then install the VIB, reboot the host then exit maintenance mode. Once that host has joined the cluster, the installer moves to the next host in the cluster and repeats the process until all hosts have the VIB installed. There is no way to install the VIBs without rebooting the ESXi hosts via the RVC method. ESXi doesn't offer any supported way to restart hostd without a reboot. 2.4.1 vsan.health.cluster_install The next step is to install the Virtual SAN Health Extension VIB on all hosts in the cluster. To complete the installation, all hosts will be rebooted in a rolling fashion. It is required for DRS to be in fully automated mode such that a rolling reboot can be performed. A rolling reboot puts one host at a time in Maintenance Mode, reboots it, and then proceeds to the next host until all hosts are rebooted. Through this rolling upgrade approach, all VMs continue to run and the reboot of the ESXi hosts does not impact production workloads. Due to this rolling reboot, the install task will take a while (minutes to hours), depending on how many host are in the cluster and how many VMs are running. If DRS is not enabled in fully automated mode, the install will not proceed. It will complain that there is an issue found with fully automated DRS check, as shown here: > vsan.health.cluster_install VSAN6-Cluster is NOT OK for installing VSAN Health Extension on ESXs. +--------------------+---------+------------+ | HostName | Version | Compatible | +--------------------+---------+------------+ | cs-ie-h02.ie.local | 6.0.0 | OK | | cs-ie-h03.ie.local | 6.0.0 | OK | | cs-ie-h01.ie.local | 6.0.0 | OK | | cs-ie-h04.ie.local | 6.0.0 | OK | +--------------------+---------+------------+ +---------------------------+-------------+ | Checks | Result | +---------------------------+-------------+ | Fully automated DRS check | Issue Found | | EAM connectivity check | OK | +---------------------------+-------------+



Another option to consider is the-d [-dry-run] option, which checks if everything is OK from an installation perspective: > vsan.health.cluster_install -d VSAN6-Cluster is OK for installing VSAN Health Extension on ESXs. A correct install will look similar to the following: > vsan.health.cluster_install This command is going to install the VSAN Health Extension VIB on all hosts in the cluster. To complete the install all hosts are going to be rebooted. It is required for DRS to be in fully automated mode such that a rolling reboot can be performed. A rolling reboot puts one host at a time in Maintenance Mode and then reboots it, then proceeding with the next until all hosts are rebooted. By doing so all VMs keep running and the reboot doesn't impact production workloads. Due to this rolling reboot the install task will take a while (minutes to hours), depending on how many host are in the cluster and how many VMs are running. Task: Install VSAN health ESX extension General event: ESX Agent Manager agency was updated New progress: 5% New status: Configuring EAM to install VIB on cluster General event: EAM reported: Host reboot is required to complete agent VIB installation General event: EAM reported: Host reboot is required to complete agent VIB installation General event: EAM reported: Host reboot is required to complete agent VIB installation General event: EAM reported: Host reboot is required to complete agent VIB installation General event: Waiting for EAM to auto-remediate New progress: 50% New status: Waiting for EAM to auto-remediate General event: Done with host esx-02a.corp.local New progress: 60% New status: 3 host(s) left to remediate General event: Done with host esx-01a.corp.local New progress: 70% New status: 2 host(s) left to remediate General event: Done with host esx-03a.corp.local New progress: 80% New status: 1 host(s) left to remediate General event: Done with host esx-04a.corp.local Task result: success



2.4.2 vsan.health.cluster_status When the installation is completed, the RVC command, vsan.health.cluster_status when run with the cluster as an argument, displays the installation status. Note: The versions shown here will be different to those used when the health check is released. > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: installed (OK) Host 'cs-ie-h04.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h01.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h03.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h02.ie.local' has health system version '6.0.0' installed vCenter Server has health system version '6.0.0' installed >

2.4.3 Changes to General View with Health check plugin Once health checks have been deployed, a few additional pieces of information are displayed in the Virtual SAN > General view.

The On-disk Format Version displays how many of the disks have been upgraded to version 2, the VirstoFS on-disk format included with Virtual SAN 6.0. Customers should upgrade to version 2 to leverage new performance and scalability improvements with snapshots and clones. The upgrade can be done online via a rolling-upgrade mechanism. Refer to the Virtual SAN Administrators Guide for the detailed steps. In the above example, all disks are at version 2. There are 0 disks with an outdated v1 on-disk format version. If disks are discovered with an outdated on-disk format, the Upgrade button will be available and may be used to update them to version 2.



2.5 Upgrading Health Check ESXi VIBs This section will not be of relevance for the initial deployment of the health check plugin, but is included for reference when future editions of the plugin are released. During an upgrade of the health check plugin, the RPM (appliance) or MSI (Windows) packages need to be updated on vCenter server. Once that step is done, and the vCenter Server services are restarted, the health checks will report that it is out of date (note that the versions shown here are pre-release):

To upgrade the hosts via the UI, this time just click on the Upgrade button. You will receive the following pop-up:

Make sure that DRS is enabled, and fully automated, on the cluster. The upgrade process follows a rolling pattern of enter maintenance mode, VIB update, reboot on a host by host basis, similar to how the install process works. This mismatch between ESXi hosts and vCenter server will also be highlighted in the actual health check tests. The test VSAN Health Service up-to-date verifies that the vCenter Server and ESXi hosts are running the same version.



2.6 Manual VIB Install If you wish to manually install the health check VIB on the ESXi hosts rather than using the UI or RVC methods, you can download it from the following location: https://my.vmware.com/group/vmware/get-download?downloadGroup=VSANHEALTH600 Refer to the official vSphere documentation on how to use esxcli software vib commands to install this VIB on ESXi hosts. 2.7 vSphere Update Manager users warning For vSphere Update Manager users, once the VIB is manually installed, or installed via RVC, a scan or compliance check is run against the ESXi host will generate vSphere Update Manager (VUM) alerts stating that the hosts are not compliant. The VIB should be installed as a vSphere Extension, not a vSphere Update.



3. Health Check Plugin UI features Once enabled, the Health service status changes to Enabled and the version of the Health check is displayed.

3.1 Enable Customer Experience Improvement Program As mentioned earlier, if you do no choose to enable the Customer Experience Improvement Program at health check install time, it can be done later from the Manage view of Health, shown below. As you can see, it is currently in a state of Disabled (not sending any data). To enable, it simply click on the Enable button:

On clicking Enable, the following popup is displayed, describing more about the CEIP and the information gathered:



Click on the hyperlink if you wish to learn more about CEIP. If you choose Yes, data collection commences, and status changed to Enabled. Participation in the CEIP can be disabled at any time.

3.2 HCL Database One of the health checks ensures that your environment uses supported hardware, software and drivers. These checks are run against a HCL file that is regularly updated, adding support for newer hardware, software and drivers, and occasionally dropping support for older hardware, software and drivers. The HCL Database buttons allows the local HCL file to be updated. This may be done directly from the Internet, or if your vCenter server is not connected to the Internet, it can be updated via a file that was downloaded from VMware, and then transferred to the vCenter server. The location of the HCL file is in section 8.5 of this document. 3.3 Support Assistant One final feature displayed in this view is the Support Assistant. This relates to the ability to upload support bundles to a Service Request opened with VMware Global Support Services (GSS). For further details on the Support Assistant, refer to the following documentation. https://www.vmware.com/support/pubs/support-assistant-pubs.html



3.4 External Proxy Settings If you do not have direct access to the Internet from your vCenter server, but you have a proxy that allows Internet connectivity, the external proxy settings can be used to provide this information to the health check plugin. Health check plugin features such as the CEIP and HCL database updates will use the proxy settings to access VMware.com via the proxy settings.



4. Health Check Plugin Uninstall If there is ever a need to uninstall the health check VIB from the ESXi hosts, the following procedures, one via the UI and the other via RVC, will detail how this can be done. 4.1 ESXi Uninstall via UI To uninstall the health services, simply click on the Disable button visible in the Health view of the UI, as shown below.

This will give you details regarding what will happen during the disable/uninstall process (it once again takes a rolling approach). Click Yes to continue.

Once the VIB has been uninstalled from each of the hosts, the health check plugin can be removed from the vCenter Server, using RPM commands on the vCenter Appliance or the Add/Remove programs feature on the Windows version.



4.2 ESXi Uninstall via vsan.health.cluster_uninstall To uninstall the Health check VIB from each of the hosts in the Virtual SAN cluster, the following RVC command should be run. This will place each host in the cluster into maintenance mode in a rolling fashion, evacuate the disk groups on each host, remove the Health Checks VIB, reboot the ESXi host and when the host is back online and rejoins the cluster, the procedure will move onto the next ESXi host in the cluster until all nodes have had the VIB uninstalled. > vsan.health.cluster_uninstall This command is going to uninstall the VSAN Health Extension VIB from all hosts in the cluster. To complete the uninstall all hosts are going to be rebooted. It is required for DRS to be in fully automated mode such that a rolling reboot can be performed. A rolling reboot puts one host at a time in Maintenance Mode and then reboots it, then proceeding with the next until all hosts are rebooted. By doing so all VMs keep running and the reboot doesn't impact production workloads. Due to this rolling reboot the uninstall task will take a while (minutes to hours), depending on how many host are in the cluster and how many VMs are running. Task: Uninstall VSAN health ESX extension New progress: 5% New status: Configuring EAM to uninstall VIB from cluster General event: EAM reported: Host reboot is required to complete agent VIB installation General event: EAM reported: Host reboot is required to complete agent VIB installation General event: EAM reported: Host reboot is required to complete agent VIB installation General event: EAM reported: Host reboot is required to complete agent VIB installation General event: Waiting for EAM to auto-remediate New progress: 50% New status: Waiting for EAM to auto-remediate General event: Done with host esx-01a.corp.local New progress: 60% New status: 3 host(s) left to remediate General event: Done with host esx-03a.corp.local New progress: 70% New status: 2 host(s) left to remediate General event: Done with host esx-04a.corp.local New progress: 80% New status: 1 host(s) left to remediate General event: Done with host esx-02a.corp.local Task result: success



Post uninstall, the status of the health check plugin can be checked as follows: > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: not installed (OK) WARNING: Host 'esx-01a.corp.local' doesn't have VIB installed WARNING: Host 'esx-02a.corp.local' doesn't have VIB installed WARNING: Host 'esx-03a.corp.local' doesn't have VIB installed WARNING: Host 'esx-04a.corp.local' doesn't have VIB installed Host 'esx-03a.corp.local' doesn't have health system installed Host 'esx-01a.corp.local' doesn't have health system installed Host 'esx-02a.corp.local' doesn't have health system installed Host 'esx-04a.corp.local' doesn't have health system installed vCenter Server has health system version '6.0' installed >

4.3 Removing Health Check plugin from vCenter Once the health check VIBs have been removed from the ESXi hosts, the next step is to remove the software from the vCenter server. 4.3.1 vCenter Appliance uninstall To do a complete cleanup of the health check plugin from a vCenter Server Linux appliance, run the command: /usr/lib/vmware-vpx/firstboot/health-rpm-uninstall.sh Here is an example of the uninstall script running on a vCenter appliance. ie-vcsa-09:~ # /usr/lib/vmware-vpx/firstboot/health-rpm-uninstall.sh Services 'vpxd' and 'vsphere-client' need to be restarted. This will disrupt all ongoing sessions. Do you want to continue? [Yes/No]? Yes /usr/lib/vmware-vpx/workflow/bin 2015-04-22T10:17:37.031Z Getting value for install-parameter: workflow.int.service-port 2015-04-22T10:17:37.040Z Getting value for install-parameter: workflow.int.jmx-port 2015-04-22T10:17:37.049Z Getting value for install-parameter: vpxd.int.sdk-port 2015-04-22T10:17:37.057Z Getting value for install-parameter: vpxd.int.sdk-tunnel-port 2015-04-22T10:17:37.066Z Getting value for install-parameter: rhttpproxy.ext.port1 2015-04-22T10:17:37.075Z Getting value for install-parameter: rhttpproxy.ext.port2 {'vpxd_sdk_tunnel_port': '8089', 'rhttpproxy_https_port': '443', 'rhttpproxy_http_port': '80', 'workflow_service_port': '8088', 'vpxd_sdk_port': '8085', 'PASSWORD': '', 'workflow_jmx_port': '19999'} 2015-04-22T10:17:37.083Z Getting value for install-parameter: syslog.ext.port 2015-04-22T10:17:37.092Z Getting value for install-parameter: vc.home.path 2015-04-22T10:17:37.100Z Getting value for install-parameter: vc.conf.path 2015-04-22T10:17:37.101Z VSAN Health service stop started 2015-04-22T10:17:37.101Z Stopping vsan-health 2015-04-22T10:17:37.101Z Running command: ['/sbin/service', 'vmware-vsan-health', 'stop'] 2015-04-22T10:17:40.168Z Done running command 2015-04-22T10:17:40.168Z Successfully stopped service vmware-vsan-health 2015-04-22T10:17:40.168Z VSAN Health stop successfully finished vmware-vpxd: Stopping vpxd by administrative request. process id was 14988 success vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd Waiting for the embedded database to start up: success Executing pre-startup scripts... vmware-vpxd: Starting vpxd by administrative request. success vmware-vpxd: Waiting for vpxd to start listening for requests on 8089



Waiting for vpxd to initialize: .success vmware-vpxd: vpxd has initialized. Stopping VMware vSphere Web Client... Stopped VMware vSphere Web Client. Last login: Wed Apr 22 10:17:56 UTC 2015 on pts/0 Starting VMware vSphere Web Client... Waiting for VMware vSphere Web Client...... running: PID:1937 ie-vcsa-09:~ #

4.3.2 Windows vCenter Server uninstall For vCenter server running on Microsoft Windows, the installer should be used to remove the health check plugin followed by a restart of the vCenter server services. The detailed steps are: Stop VMware vSphere Web Client service

o From the "Start" menu, select "Computer Management->Service and Applications->Services" o In the "Services" windows, find a service named "VMware vSphere

Web Client", right click "Stop" o Notice that it will also stop a service named "VMware Performance Charts" as well

Uninstall Virtual SAN health service from Windows service management o From the "Start" menu, select "Control Panel", then click "Uninstall a program" menu o Browse and find the entry of "VMware VSAN health service" and click "Uninstall" menu

Restart vSphere VirtualCenter Server and VMware vSphere Web Client service o From "Start" menu, select "Computer Management -> Service and Applications -> Services" o In the "Services" window, find a service named "VMware VirtualCenter

Server", right click "Restart" o In the "Services" windows, find a service named "VMware vSphere

Web Client", right click "Start"



5. Health check via the CLI

5.1 Health check overview Once the Health check plugin is installed, they can be used for verifying the health of the Virtual SAN environment. The first step is to verify that the health checks are enabled. Once they are enabled, the individual checks can be examined to see if there are any issues with this Virtual SAN deployment. The health checks are split into five distinct categories: 1. Cluster health 2. Network health 3. Data health 4. Limits health 5. Physical disks health Each of these categories has a number of individual checks, which will be looked at in detail shortly. The health check plugin provides checks at both the command line and the user interface. The command line checks are part of RVC, the Ruby vSphere Console. The first of these health checks is simply a status check to ensure that the health check has been successfully installed. The full list of RVC health check commands are provided here: vsan.health.cluster_attach_to_sr vsan.health.cluster_customer_experience_program_configure vsan.health.cluster_customer_experience_program_status vsan.health.cluster_debug_multicast vsan.health.cluster_install vsan.health.cluster_load_test_cleanup vsan.health.cluster_load_test_prepare vsan.health.cluster_load_test_run vsan.health.cluster_proxy_configure vsan.health.cluster_proxy_status vsan.health.cluster_repair_immediately vsan.health.cluster_status vsan.health.cluster_uninstall vsan.health.hcl_update_db vsan.health.health_summary vsan.health.multicast_speed_test



All of the commands can be run with a h (help) option for further information. Many of these commands are related to proactive tests, which will be looked at in detail in chapter 7 of this guide. 5.1 Check status - vsan.health.cluster_status Here is an example of what a successful installation looks like: > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: installed (OK) Host 'cs-ie-h04.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h01.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h03.ie.local' has health system version '6.0.0' installed Host 'cs-ie-h02.ie.local' has health system version '6.0.0' installed vCenter Server has health system version '6.0.0' installed > In the next check, it would appear that a host reboot is required on host cs-ie-h01 to complete the VIB installation (ignore the versions shown here as these were captured from a pre-release version of health check): > vsan.health.cluster_status Configuration of ESX VSAN Health Extension: installed (incomplete, see issues) Issues: Host reboot is required to complete agent VIB installation Per-Host details: Host 'cs-ie-h04.ie.local': Status: green Host 'cs-ie-h03.ie.local': Status: green Host 'cs-ie-h01.ie.local': Status: red Issues: Host reboot is required to complete agent VIB installation Host 'cs-ie-h02.ie.local': Status: green Host 'cs-ie-h04.ie.local' has health system version '0.4' installed Host 'cs-ie-h01.ie.local' has health system version '0.4' installed Host 'cs-ie-h02.ie.local' has health system version '0.4' installed Host 'cs-ie-h03.ie.local' has health system version '0.4' installed vCenter Server has health system version '0.4' installed

How to Use Black Cat Usb

Documents

Transcript of How to Use Black Cat Usb