TROUBLESHOOT ISSUES GATHERING CLUSTER LOGS · 7 - EMC Isilon Customer Troubleshooting Guide:...

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.

___________________________

1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs

___________________

Abstract This guide helps you troubleshoot issues using the isi_gather_info command to gather and upload cluster logs.

August 13, 2019

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT ISSUES GATHERING CLUSTER LOGS

OneFS 7.2 - 8.2

https://community.emc.com/docs/DOC-49017

http://bit.ly/isilon-docfeedback



___________________________


___________________

Contents and overview

Before you beginPage 3

Appendix A If you need further assistance

Start troubleshootingPage 4

RoutingPage 5

Issues gathering log filesPage 6

Upload logs using HTTPPage 10

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these steps.

2. Perform troubleshooting steps in order.

Appendix BHow to use this flowchart

Upload logs using FTPPage 13

Debug log gatherPage 15

Appendix CExample output

3. Appendixes





___________________________


___________________

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.

Configure screen logging through SSHWe recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with EMC Isilon Technical Support, if you require assistance at any point during troubleshooting.

1. Open an SSH connection to the clsuter and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.

2. Change the directory to /ifs/data/Isilon_Support by running the following command:

cd /ifs/data/Isilon_Support

3. Run the following command to capture all input and output from the session:

screen -L

This will create a file named screenlog.0 that will be appended to during your session.

4. Perform troubleshooting.



https://support.emc.com/kb/304071



___________________________


___________________

Start troubleshooting

Did you receive a Per mi ssi on deni ed

error?

Start

IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.

If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on page 3.

No

Are you having an issue gathering

logs?

Yes

Refer toOneFS: when running isi_gather_info you get Permission denied, article 499425

Go to Page 5

Go to Page 6Yes

No

Is the output from i si _gat her _i nf o

showing that all steps have been SKI PPED?

No

Refer toIsilon OneFS 8.2 isi_gather_info does not run and you see SKIPPED across each line item, article 535679

YesExample outputConf i gur i ng SKI PPEDr unni ng l ocal commands SKI PPEDr unni ng node commands SKI PPEDcol l ect i ng f i l es SKI PPEDgener at i ng package_i nf o. xml SKI PPEDt ar r i ng gat her SKI PPEDupl oadi ng gat her SKI PPEDcl eani ng up I N PROGRESS \







___________________________


___________________

Routing

You could have arrived here from:

- Page 4 - Start troubleshooting

Are you having an issue uploading log

files?

No

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support, as instructed in

Appendix A.

Yes

Which method is being used to transfer logs?

Go to Page 10 Go to Page 13

HTTP

FTP

ESRS





___________________________


___________________

Issues gathering log files


- Page 4 - Start troubleshooting

Are you using the OneFS web administration

interface?

YesWe suggest using the

command-line interface (CLI) for all log gathers.

Are you using options such as

- - i ncr ement al , - a, - A, etc.?

No

No

Did the log gather complete

successfully?

YesRun the following command

without any options:

isi_gather_info

YesEnd troubleshooting


NoYes, but withperformance

issues



___________________________


___________________

Issues gathering log files (2)


- Page 6 - Issues gathering log files

Verify the health of the nodes in your cluster by running:

isi status

The Heal t h DASR column will indicate node health. See Appendix C for example output.

Are all nodes healthy?

Go to Page 9Yes

No

Run the following command and exclude the unhealthy nodes, where < healthy> is the list of nodes that have a status of OK:

isi_gather_info -n '< healthy> '

Using the output in Appendix C as an example, the command would look like this:i si _gat her _i nf o - n ' 2'

The - n flag indicates that the command will gather information only from the listed nodes. The list must be a comma-separated list or range of logical node numbers (LNNs), which is indicated in the ID column of the i si st at us output. If no nodes are explicitly listed,

the command gathers information from all cluster nodes. Note that nodes are automatically excluded if they are marked as Down.

Did the command complete

successfully?Go to Page 9 Go to Page 8YesNo

__________

_________



___________________________


___________________



- Page 7 - Issues gathering log files

This indicates that the problem is the unhealthy nodes. On the nodes that have a status of At t ent i on, run the

following command and try to resolve any issues that are presented:

isi events

Once you have cleared up any issues, run:

isi_gather_info

Did the log gather complete

successfully?YesEnd troubleshooting


NoYes, but withperformance

issues



___________________________


___________________



- Page 7 - Issues gathering log files (2)- Page 8 - Issues gathering log files (3)

Does the log gather seem to be

hanging?No

Review the /var/log/messages log for error messages. Search the knowledge base for

known issues relating to any error messages that you find in the log. Try to resolve issues

with any articles that you find.

Yes

Run this command to see which nodes are still processing the log gather:

isi_for_array "ps auxww | grep _gather | grep -v grep"

For every node that is still processing the log gather, open an SSH connection to that node and run the following command, where

< directory> is the name of the directory that resulted from running the previous command. The directory name changes every time you

run the command:

bash -x /ifs/data/Isilon_Support /tm p/< directory> /_gather



Appendix A.

Were you able to find a

solution?

End troubleshooting

No

Yes

____________________________________________________________

___________________________


___________________

Upload logs using HTTP


- Page 5 - Routing

Are you using the OneFS web administration

interface?

YesWe suggest using the

command-line interface (CLI) for all log gathers.

No

Find the ht t p_pr oxy_host line and note whether an IP address or host name is

listed there. If there is a value listed, that indicates that a proxy server is being used.

Go to Page 11

Run the following command to verify whether an HTTP proxy server is being used:

grep ht tp_ /etc/m cp/override/gather.xm l

If there is a proxy server, the output will appear similar to the following:

<set t i ng name=" ht t p_enabl e" >Tr ueTr ue</ set t i ng><set t i ng name=" ht t p_host " >178. 178. 178. 221</ set t i ng><set t i ng name=" ht t p_pat h" >/ dat a</ set t i ng><set t i ng name=" ht t p_pr oxy_host " >178. 178. 178. 222</ set t i ng><set t i ng name=" ht t p_pr oxy_por t " >1313</ set t i ng>

___________________________


___________________

Upload logs using HTTP (2)


- Page 10 - Upload logs using HTTP

Is a proxy server being

used?Go to Page 12

Run the following command, where < ht tp_proxy_host> and < ht tp_proxy_port> are the values from the output in the previous step:

export ht tp_proxy= ht tp://< ht tp_proxy_host> :< ht tp_proxy_port>

For example:expor t ht t p_pr oxy=ht t p: / / 178. 178. 178. 222: 1313

This command sets a variable in the current shell session so when it tries to upload it will use the specified proxy.

No

Yes

Go to Page 12

___________________________


___________________

Upload logs using HTTP (3)


- Page 11 - Upload logs using HTTP (2)

Run

isi_gather_info

The output of the command gives you the path to the gathered log set on your cluster and will look similar to the following:

Packagi ng compl et e. . . Package: / i f s/ dat a/ I si l on_Suppor t / pkg/ I si l onLogs- Cl ust er 1- 20190611- 115485. t gz

Perform a test upload using the curl command to reproduce the upload that previously failed when you ran i si _gat her _i nf o. Run the command, where:

- < path to log file> is the same path to the gathered log set obtained from the output of i si _gat her _i nf o above.

- < ht tp_host> and < ht tp_path> are the values specified in the output of the grep ht tp_ /etc/m cp/override/gather.xm l command you ran on page 11.

- < log file nam e> is the name of the file from < path to log file> .

curl -T < path to log file> "ht tp://< ht tp_host> /< ht tp_path> /< log file nam e> "

Using the output from previous pages, here is an example:

cur l - T / i f s/ dat a/ I si l on_Suppor t / pkg/ I si l onLogs- Cl ust er 1- 20190611- 115485. t gz ht t p: / / 178. 178. 178. 221/ dat a/ I si l onLogs- Cl ust er 1- 20190611- 115485. t gz



Appendix A.



___________________________


___________________

Upload logs using FTP


- Page 5- Routing- Page 14 - Upload logs using FTP (2)

Run the following command to upload logs using FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support

Was the upload

successful?Yes End troubleshooting

No

Refer toHow to upload files to Isilon Technical Support, article 304567

Go to Page 14

__________________________________________






___________________________


___________________

Upload logs using FTP (2)


- Page 13 - Upload logs using FTP

Is this the second time you have been at this point in

this guide?

YesThis indicates a possible issue with your environment. Engage

your FTP administrators to troubleshoot the issue further.

Change FTP to passive mode by running the following command:

export ftp_passive_m ode= yes

Note Changing FTP to passive mode only persists through the current SSH session. The default mode is active mode. If you wish to change it back to active mode before the SSH session ends, run:

export ftp_passive_m ode= no

No

Return to Page 13





___________________________


___________________

Debug log gather


- Page 6 - Issues gathering log files- Page 8 - Issues gathering log files (3)

Debug while gathering logs by running the following command:

isi_gather_info --debug --verbose

This log gather will help Isilon Technical Support determine the source of your issue.



Appendix A.

_________________________________________________________





___________________________


___________________

Appendix A: If you need further assistance

Contact Isilon Technical SupportIf you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Upload node log files and the screen log file to EMC Isilon Technical Support1. When troubleshooting is complete, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your method of uploading files. If you are not sure which method to use, use FTP.

ESRS: isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support /screenlog.0

FTP: isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support /screenlog.0

HTTP: isi_gather_info --ht tp --local-only -f /ifs/data/Isilon_Support /screenlog.0

SupportIQ:Copy and paste the following command.Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support /screenlog.0 --noupload \--sym link /var/crash/Support IQ/upload/ftp

3. If you receive a message that the upload was unsuccessful, refer to article 304567 for directions on how to upload files over FTP.

____________



http://www.emc.com/contact/contact-us.htm




___________________________


___________________

Decision diamondYes No

Process stepProcess step with command:

com m and xyz

Go to Page #

Page # Note

Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.

CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.

End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.

Optional process step

IntroductionDescribes what the section helps you to accomplish.

Appendix B: How to use this flowchart


- Page 4 - Start Troubleshooting

Directional arrows indicate the path through

the process flow.



___________________________


___________________

Appendix C: Example output


- Page 7 - Issues gathering log files (2)

Example outputStatus of - A- - indicates At t ent i on.

cl ust er - 1# i si st at usCl ust er Name: cl ust erCl ust er Heal t h: [ ATTN]Cl ust er St or age: HDD SSD St or ageSi ze: 17. 4T ( 18. 1T Raw) 0 ( 0 Raw)VHS Si ze: 644. 6GUsed: 4. 4G ( < 1%) 0 ( n/ a)Avai l : 17. 4T ( > 99%) 0 ( n/ a) Heal t h Thr oughput ( bps) HDD St or age SSD St or ageI D | I P Addr ess | DASR | I n Out Tot al | Used / Si ze | Used / Si ze- - - +- - - - - - - - - - - - - - - +- - - - - +- - - - - +- - - - - +- - - - - +- - - - - - - - - - - - - - - - - +- - - - - - - - - - - - - - - - - 1| 11. 22. 33. 44 | - A- - | 0| 22. 7k| 22. 7k| 1. 4G/ 5. 7T( < 1%) | L3: 186G 2| 11. 22. 33. 44 | OK | 0| 0| 0| 1. 5G/ 6. 0T( < 1%) | L3: 186G 3| 11. 22. 33. 44 | - A- - | 0| 77. 7k| 77. 7k| 1. 4G/ 5. 7T( < 1%) | L3: 186G- - - +- - - - - - - - - - - - - - - +- - - - - +- - - - - +- - - - - +- - - - - +- - - - - - - - - - - - - - - - - +- - - - - - - - - - - - - - - - -Cl ust er Tot al s: | 0| 100k| 100k| 4. 4G/ 17. 4T( < 1%) | L3: 559G Heal t h Fi el ds: D = Down, A = At t ent i on, S = Smar t f ai l ed, R = Read- Onl y



___________________________


___________________

Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com



https://www.emc.com

TROUBLESHOOT ISSUES GATHERING CLUSTER LOGS · 7 - EMC Isilon Customer Troubleshooting Guide:...

Documents

Transcript of TROUBLESHOOT ISSUES GATHERING CLUSTER LOGS · 7 - EMC Isilon Customer Troubleshooting Guide:...