Tips and Tricks with vCenter Log Insight (NEW!)

Michael White, VMware

VCM4528

#VCM4528

2

Problem: Operate and Troubleshoot a Complex System

VMware Logs

OS and

App Logs

200 ESXi Host + VMs = 200GB or 2B log events per day

Physical Infrastructure Logs

3

4

Introducing VMware vCenter Log Insight

VMware’s New Log Analytics Solution

• Make sense of all your log data

• Best for vSphere logs, extensible to OS, app,

storage and networking device logs

• Easy-to-use virtual appliance

• Simple and predictable pricing model

Key Use Cases

• IT Operations – Troubleshooting, Monitoring,

Root Cause Analysis

• Security Monitoring, Compliance, Business

Transaction Monitoring, …

Available Now!

• 60-day Trial: www.vmware.com/try-vmware

5

Agenda

Install

Configure

Reporters

Tagging

Content Pack

Scalability

Examples

Demo

Miscellaneous

The End and Thank you!

(Appendix)

6

Install Tidbits

Use FQDN for name during deploy

Before power on, add disk

Add 100 GB to start and figure out what you need (we’ll help)

Have at least one source configured before install

No spelling checker in the Network info area – double check!

Data-core should be what you added + 97GB – this is storage

for events

7

Configure

Once installed, we need to configure for use

Before you start configuring, change root password at console –

this will enable SSH support.

8

Configure – Continued

Now connect to the vC Log Insight URL

9

Configure – Continued

10

Configure – Continued

Add your license and use the Set Key button

11

Configure – Continued

12

Configure – Continued

13

Configure – Continued

14

Configure – Continued

15

Configure – Continued

16

Configure – Continued

17

Configure – Continued

18

Sources

Whole stack is key!

Storage – some easier than others

Networking – Cisco, vCNS – both easy

ESX(i) – easy

vCenter (vC) – harder

vCenter Server Appliance (vCSA) – easy but with a catch

View – can send only events but not not anything else – so treat

like Windows vC

Things to know

• Links in Appendix

• ESXi stops reporting when interrupted – needs attention

19

Sources – Continued

Things to know – Continued

• Windows is harder – need to use a forwarder – I use Datagram

• When using a forwarder log location is key – Check Appendix for locations

20

Sources – Continued

21

Tagging

Important for when you have one host or VM with many log files

being sent to LI

Doing a search will normally search all of the log files from a host

If you use tagging, you can do a search on host AND tag, and

assuming one tag per log file you can do a much more granular

search which is quicker and more applicable

22

Tagging – Continued – No Tagging – on a vCSA

# vpxd source log

source vpxd {

file("/var/log/vmware/vpx/vpxd.log" follow_freq(1) flags(no-parse));

file("/var/log/vmware/vpx/vpxd-alert.log" follow_freq(1) flags(no-parse));

file("/var/log/vmware/vpx/vws.log" follow_freq(1) flags(no-parse));

file("/var/log/vmware/vpx/vmware-vpxd.log" follow_freq(1) flags(no-parse));

file("/var/log/vmware/vpx/inventoryservice/ds.log" follow_freq(1) flags(no-parse));

};

# Remote Syslog Host

destination remote_syslog {

udp("a.b.c.d" port (514));

;

# Log vCenter Server vpxd log remotely

log {

source(vpxd);

destination(remote_syslog);

};

23

Tagging – Continued

So using the tags looks like:

file("/var/log/vmware/vpx/vpxd.log" follow_freq(1) log_prefix(“VC_APP: “) flags(no-parse));

file("/var/log/vmware/vpx/vpxd-alert.log" follow_freq(1) log_prefix(“VC_ALERT: “) flags(no-parse));

file("/var/log/vmware/vpx/vws.log" follow_freq(1) log_prefix(“VC_VWS: “) flags(no-parse));

file("/var/log/vmware/vpx/vmware-vpxd.log" follow_freq(1) log_prefix(“VC_VMW_VPX: “) flags(no-parse));

file("/var/log/vmware/vpx/inventoryservice/ds.log" follow_freq(1) log_prefix(“VC_IS: “) flags(no-parse));

24

Tagging – Continued – Normal

So when using the tags to search looks like:

;

25

Content Packs

A Content Pack provides best practices and

knowledge about the logs

It consists of: Queries, alerts, dashboards

and field extractions

VMware and our partners are working on

Content Packs

vSphere Content Pack

• Ships out of the box

• Knowledge about ESXi and vCenter Server logs as

well as vC Alarms, Events & Tasks

• It consists of: Queries, alerts, dashboards and field

extractions

• Divided into functional categories

• ESX, Storage and vCenter including Alarms

• vSphere and Content Pack dashboards are NOT

editable – users can clone them into their workspace

26

Content Packs – Continued

27

Content Packs – Continued

28

Content Packs – Continued

29

Announcing the Log Insight Content Pack Market Place

And more…

https://solutionexchange.vmware.com/store/loginsight

Extend vCenter Log Insight with Content Packs from:

30

Scalability – Guidelines

Watch ‘outside’ of VM with your normal tools, i.e vC

Operations Manager

Watch ‘inside of vC LI with Health \ System Info

31

Scalability – Guidelines – Continued

32

Scalability – Guidelines – Continued

33

Scalability – Guidelines – Storage

In case we misjudge on storage, enable Data Archiving

Remember that events, once in vC LI are rotated out as disk space

usable is reduced – either to trash or Data Archiving (system alert)

– first in, and first out

If you have to import archived events, than use new instance of LI!

Rough guide – 250 MB per day per ESX host, and 50 MB per day

for other devices – retention time is decided by available storage

and archiving

34

Scalability – Guidelines – Storage

You can enable Data Archiving on the Storage window in

Administration. Once enabled you will be alerted when Archiving

is about to occur. At that time can add disk or not!

35

How Much Disk Space for 30 Days Retention?

Gross estimate:

267 bytes/message

This example:

23*267*60*60*24*30

= ~16GB per

30 days

More accurate estimations can be found in runtime.log

During failures, log volume will increase significantly

• Overprovision!

36

Examples – Bad Credentials

37

Examples – High Latency by Host

38

Miscellaneous

Support Log

• UI – On the Health page of Settings Administration

• CLI – log in on console and execute loginsight-support

• With every support call!

Backup

• VDP, VDPA, etc.

• Image

vC Ops

• Launch in Context

39

Miscellaneous

vC Ops

• Launch in Context

40

Miscellaneous – Continued – Alerts

• vC Ops option requires the

integration enabled and

email requires SMTP

• User alerts are different

from system alerts

• The admin cannot disable

individual alerts

41

Miscellaneous – Continued

Upgrades / Updates

• Will be a short outage

• In-place which makes it easy

• Get .rpm same place on vmware.com you got .ova

• SCP update file to LI in /tmp and execute with rpm –Uvh file_name

• Than test and check Settings \ About for new version – does it match?

42

Miscellaneous – Continued

Fixing IP issues

• Not too hard but tricky – is

best to get it right!

• Install again correctly is

great choice

• vApp modifications is other

choice – make sure VM is off,

and than Edit Settings

– vApp Options

• Not aware of any other

safe alternatives!

43

Demo

44

Summary

Source(s) working first

Add disk at the beginning to avoid outages

Ensure SMTP / vC / vC Operations connections good!

Set a good system email address destination

Monitor disk / processor carefully at first

Use Data Archiving

Most important – make sure your entire stack is reporting

Update as often as you can!

45

Other VMware Activities Related to This Session

HOL: VMware Log Insight

VMware Booth: VMware Cloud Operations

Breakout Session: Deep Dive Wed, 10-11 VCM4445

Group Discussions: Wed, 2-3 Log Insight with Steve Flanders

5 Free License Trial available when you follow @vmLogInsight

HOL:

HOL-SDC-1301

VMware vCenter Log Insight - Unchained from the Allegory

Group Discussions:

VCM1005-GD

Log Insight with Steve Flanders

THANK YOU

Tips and Tricks with vCenter Log Insight (NEW!)

Michael White, VMware

VCM4528

#VCM4528

49

Appendix

Links

• Configuring Remote Syslog on VMware products -

http://sflanders.net/2013/06/24/configure-remote-syslog-on-vmware-products/

• Datagram - http://www.syslogserver.com/syslogagent.html

• Release notes - http://www.vmware.com/support/log-insight/doc/log-insight-10-

release-notes.html

• NetApp syslog - https://communities.netapp.com/docs/DOC-5048

• vCloud Suite - http://www.virtuallyghetto.com/2013/06/forwarding-logs-from-

vcloud-suite-to.html - includes a script to help which include tagging!

• ESXi, syslog, and logins – great blog about how to capture logins – of different

types in ESXi - http://blogs.vmware.com/vsphere/2013/07/capturing-logins-to-

esxi-by-a-root-account.html

• Symmetrix - http://codyhosterman.com/2013/07/10/using-vmwares-vcenter-log-

insight-with-symmetrix-vmax/

50

Appendix – Continued

Links – Continued

• Detecting stopped ESXi syslog forwarding -

http://www.virtuallyghetto.com/2012/07/detecting-esxi-remote-syslog-

connection.html - important, and I suggest using script option

• VM Monitoring log forwarding - http://www.virtuallyghetto.com/2013/07/a-hidden-

vsphere-51-gem-forwarding_10.html

• Install and Admin Guide - http://www.vmware.com/pdf/log-insight-10-install-

admin-guide.pdf

• Users Guide - http://www.vmware.com/pdf/log-insight-10-users-guide.pdf

• Security Guide - http://www.vmware.com/pdf/log-insight-10-security-guide.pdf

• Sample for firewall - http://www.virtualclouds.co.za/?p=740

• Sending Alerts to vC Ops - http://www.virtualclouds.co.za/?p=771

• Location of log files for VMware products – http://kb.vmware.com/kb/1021806

• LI community - http://loginsight.vmware.com

• Try it out - http://www.vmware.com/go/try-log-insight

51

Architecture Overview: Log Insight Deployment Option 1

Considerations:

• Good for log

management greenfield

• Less flexible as syslog-ng

can split the logs into

multiple destinations (e.g.

one to syslog one to local

disk) but LI cannot. Some

senders might still be able to

split reporting

• One UI for everything!

• Easy

ESXi

#1

ESXi

#2 … ESXi

#n

No syslog-ng/rsyslog

Log

Insight

Windows

Epilog or Datagram Syslog

Agent for file-to-syslog

52

Architecture Overview: Log Insight Deployment Option 2

Considerations:

• Requires managing

another syslog server

• More flexible as syslog-ng

can split the logs into

multiple destinations (e.g.

one to syslog one to

local disk)

• For large installations can be

more scalable as you can

have multiple levels of

rollups (e.g. one for each

“pod” or datacenter)

ESXi

#1

ESXi

#2 … ESXi

#n

Syslog

relay

Using a syslog-ng/rsyslog relay

Log

Insight

Windows

Epilog or Datagram Syslog

Agent for file-to-syslog

53

Appendix – Continued

Install Outline

Working in vSphere Web Client

54

Appendix – Continued

Install Outline – Continued

vSphere Web Client doesn’t see .ova by default (.ovl) so you need

to switch to see it – should be different soon – maybe!

55

Appendix – Continued

Install Outline – Continued

Most Important – use fully qualified domain name!

56

Appendix – Continued

Install Outline – Continued

Make sure to have enough space for now, and room to grow!

57

Appendix – Continued

Install Outline – Continued

No spelling checker here – get it all right!!

58

Appendix – Continued

Install Outline – Continued

No power on, as we need to adjust disk to start

59

Appendix – Continued

Install Outline – Continued

60

Appendix – Continued

Install Outline - Continued

61

Appendix – Continued

Install Outline – finished!