Troubleshooting JIRA and Confluence!

Jeff Curry!!

While keeping your sanity!

Atlassian Support!

Agenda!•  Troubleshooting process!•  Reviewing logfiles!

•  Reading thread dumps!

•  Taking it further!•  Getting support ( quicker )!!

Case Review!

Case Review!•  Entitled ‘Error Message’!•  Daily crashes, poor

performance!•  Larger instance!

Case Review – Reviewing Logfiles!

logs/atlassian-jira.log!( NOT catalina.out )!

QuartzWorker-0 ERROR ServiceRunner xxx [service.services.mail.MailFetcherService] xxx[10030]: Error connecting to host ‘xxx' as user ‘xxx' via protocol 'pop3s': javax.mail.MessagingException: Connect failed;nested exception is:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failedjavax.mail.MessagingException: Connect failed;nested exception is:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed!

org.apache.jk.core.MsgContext action!WARNING: Error sending end packet!java.net.SocketException: Software caused connection abort: socket write error!at java.net.SocketOutputStream.socketWrite0(Native Method)!at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)!at java.net.SocketOutputStream.write(SocketOutputStream.java:136)!at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)!at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)!at org.apache.jk.core.MsgContext.action(MsgContext.java:293)!at org.apache.coyote.Response.action(Response.java:182)!at org.apache.coyote.Response.finish(Response.java:304)!at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)!at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)!at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)!at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)!at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)!at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)!at java.lang.Thread.run(Thread.java:534)!

DefaultIndexManager - Lock attempt timed out - waited 30000 millisecondscom.atlassian.jira.issue.index.IndexException: Lock attempt timed out - waited 30000 milliseconds

!at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:463)!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:286)!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:252)!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:260)!at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)!at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)!at java.lang.reflect.Method.invoke(Method.java:618)!at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)!at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)!at $Proxy4.reIndexIssueObjects(Unknown Source)!at com.atlassian.jira.event.listeners.search.IssueIndexListener.reIndex(IssueIndexListener.java:138)!at com.atlassian.jira.event.listeners.search.IssueIndexListener.issueCommented(IssueIndexListener.java:78)!at com.atlassian.jira.event.issue.AbstractIssueEventListener.workflowEvent(AbstractIssueEventListener.java:118)!at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:125)!at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:83)!at com.atlassian.jira.web.action.issue.AddComment.alertSystemOfComment(AddComment.java:101)!at com.atlassian.jira.web.action.issue.AddComment.doExecute(AddComment.java:85)!at webwork.action.ActionSupport.execute(ActionSupport.java:153)!at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:54)!at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:132)!at com.atlassian.jira.web.dispatcher.JiraServletDispatcher.service(JiraServletDispatcher.java:178)!at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

!

Case Review – Reviewing Logfiles!

Garbage collection ( GC ) logs!-verbose:gc -Xloggc:<home>/logs/gc.log - XX:

+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintGCDateStamps!

![ http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html ]!

Case Review – Reviewing Logfiles!•  Plain!48.630: [GC [PSYoungGen: 112384K->5460K(131072K)] 112384K->5460K(1029888K), 0.0158160 secs] [Times: user=0.03 sys=0.01, real=0.03 secs] !

93.865: [GC [PSYoungGen: 117844K->16468K(131072K)] 117844K->16468K(1029888K), 0.0329740 secs] [Times: user=0.06 sys=0.03, real=0.03 secs] !

105.728: [GC [PSYoungGen: 128852K->18302K(131072K)] 128852K->18302K(1029888K), 0.0287090 secs] [Times: user=0.07 sys=0.01, real=0.02 secs] !

122.928: [GC [PSYoungGen: 130686K->18679K(131072K)] 130686K->21123K(1029888K), 0.0221030 secs] [Times:

user=0.06 sys=0.01, real=0.02 secs]!•  Filtered! grep 'Full GC' gc.log | awk '{print $1, $16}'!

9285.005: real=7.80!

9474.161: real=10.99!

10106.270: real=5.57!

10355.367: real=12.84!

10758.017: real=15.17!

Case Review - Workaround!•  GC tuning for stability!•  Concurrent !

•  -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:SurvivorRatio=8!

•  2 GB allocation!

•  Added vCPU!

!

Case Review – Thread dumps!"QuartzWorker-0" prio=10 tid=0x0a2ac400 nid=0xe3e runnable [0x72f04000]!

java.lang.Thread.State: RUNNABLE!

at java.net.SocketInputStream.socketRead0(Native Method)!

at java.net.SocketInputStream.read(SocketInputStream.java:129)!

at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)!

at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)!

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)!

- locked <0x744f7a38> (a java.lang.Object) !

Case Review – Thread dumps!at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)!

at java.io.BufferedInputStream.read(BufferedInputStream.java:237)!

- locked <0x74501140> (a java.io.BufferedInputStream)!

at java.io.DataInputStream.readLine(DataInputStream.java:496)!

at com.sun.mail.pop3.Protocol.simpleCommand(Protocol.java:360)!

at com.sun.mail.pop3.Protocol.<init>(Protocol.java:104)!

at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:214)!

- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!

at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:157)!

- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!

Case Review – Thread dumps!at javax.mail.Service.connect(Service.java:288)!

- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!

at com.atlassian.jira.service.services.mail.MailFetcherService.run(MailFetcherService.java:146)!

at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)!

at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)!

at org.quartz.core.JobRunShell.run(JobRunShell.java:195)!

at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)!

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)!

Case Review – Thread dumps!"IssueIndexer:thread-3" prio=10 tid=0x08ec1000 nid=0x1b64 runnable [0x71f44000]!

java.lang.Thread.State: RUNNABLE!

...!

at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java)!

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)!

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)!

at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:92)!

at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)!

Case Review – Thread dumps!at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:605)!

at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)!

at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:790)!

at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:775)!

at org.ofbiz.core.entity.GenericDelegator.getRelated(GenericDelegator.java:1211)!

at org.ofbiz.core.entity.GenericDelegator.getRelatedByAnd(GenericDelegator.java:1169)!

at org.ofbiz.core.entity.GenericValue.getRelatedByAnd(GenericValue.java:260)!

at com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssue(DefaultIssueManager.java:318)!

Case Review – Thread dumps!at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(DefaultCommentManager.java:135)!

at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:40)!

at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:29)!

at com.atlassian.jira.issue.index.DefaultIssueIndexer$CommentDocumentBuilder.get(DefaultIssueIndexer.java:460)!

at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:315)!

at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:371)!

!

!

Case Review - Resolution!

JIRA POP service

Issue Key

Database Mail inbox

Case Review - Resolution!

Taking it further!

Taking it further - VisualVM!•  http://visualvm.java.net!•  Monitor performance!

•  Profile memory usage!

•  Review thread dumps!

•  Browse heap dumps!

Taking it further – OS Monitoring!Linux 2.6.18-164.11.1.el5PAE (test.jira.com) !05/17/2012!

!

12:00:01 AM CPU %user %nice %system %iowait %steal %idle!

12:10:01 AM all 5.22 0.00 1.50 6.50 0.00 86.79!

12:20:01 AM all 3.42 0.00 0.90 0.08 0.00 95.60!

12:30:01 AM all 4.77 0.00 1.01 0.29 0.00 93.92!

12:40:01 AM all 3.47 0.00 0.90 0.07 0.00 95.56!

12:50:01 AM all 3.43 0.00 1.13 0.07 0.00 95.38!

01:00:01 AM all 3.48 0.00 1.12 0.06 0.00 95.33!

01:10:01 AM all 4.27 0.00 0.95 1.74 0.00 93.04!

01:20:01 AM all 3.48 0.00 1.14 0.06 0.00 95.33!

01:30:01 AM all 3.80 0.00 0.95 0.06 0.00 95.19!

01:40:01 AM all 3.43 0.00 0.90 0.07 0.00 95.60!

01:50:01 AM all 3.41 0.00 0.92 0.07 0.00 95.61!

02:00:01 AM all 3.42 0.00 0.91 0.19 0.00 95.49!

02:10:01 AM all 3.71 0.00 0.94 3.59 0.00 91.76!

02:20:01 AM all 3.38 0.00 0.90 0.08 0.00 95.65!

02:30:01 AM all 3.77 0.00 0.94 0.07 0.00 95.23!

02:40:01 AM all 3.38 0.00 0.89 0.07 0.00 95.66!

02:50:01 AM all 3.37 0.00 0.89 0.06 0.00 95.69!

03:00:01 AM all 3.37 0.00 0.89 0.07 0.00 95.67!

03:10:01 AM all 3.78 0.00 1.04 1.56 0.00 93.62!

03:20:01 AM all 3.41 0.00 0.89 0.06 0.00 95.65!

03:30:01 AM all 3.82 0.00 0.95 0.06 0.00 95.17!

03:40:01 AM all 5.94 0.00 1.43 1.33 0.00 91.30!

03:50:01 AM all 3.42 0.00 0.89 0.05 0.00 95.63!

04:00:01 AM all 3.47 0.00 0.89 0.07 0.00 95.57!

04:10:01 AM all 29.31 0.00 3.65 22.59 0.00 44.45!

Taking it further - SAR!http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html!

Taking it further – HTTP inspection!•  Fiddler!•  http://www.fiddler2.com/fiddler2/!

•  Live HTTP Headers ( Firefox addon )!•  https://addons.mozilla.org/en-US/firefox/addon/live-http-headers/!

Taking it further – HTTP inspection!

Taking it further – HTTP inspection!

Getting support ( quicker )!

Getting support – self-help!

Getting support – self-help!•  Knowledge base articles!•  https://confluence.atlassian.com/display/KB/Knowledge+Base+Home!

•  Answers.atlassian.com!•  #atlassian on irc.freenode.com!

Getting support – self-help!

!!

Getting support – Atlassian view!•  Initial response time SLA!•  1 – 1 hour!

•  2 – 4 hours!

•  3 – 8 hours!

•  4 – 24 hours!

!

Getting support – Atlassian view!

Open Waiting for Customer

Atlassian Investigating

Waiting for Customer

IRT

Getting support – file a ticket!•  When I… I expect… however…!•  Application log files !

•  Screenshots!

•  Steps to reproduce!

Getting support – file a ticket!•  When I use JIRA and try to login!•  I expect JIRA to display a login page!•  however I get this 404 page!•  log files attached!

Getting support – Atlassian view!

Open Atlassian Investigating

Waiting for Customer

IRT

Summary!•  Basic!•  Application and GC log files!

•  Thread dumps!

•  Advanced!•  VisualVM!

•  HTTP inspection!

#atlassiansummit

I am my own Atlassian support engineer AMAA

Thank you!