#ATM16

Adapting to evolving user, security, and business needs with Aruba ClearPassTroy ArnoldJohn CoxRajesh RamireddyMarch 9th, 2016 @ArubaNetworks |

2#ATM16

Proactive Monitoring

@ArubaNetworks |

3#ATM16

Dashboard Overview–Dashboard Layout and Widgets allow administrators to view summarized data

in efficient way.

@ArubaNetworks |

4#ATM16

New Dashboard Widgets

– Endpoint Profiler Summary– MDM Discovery Summary– OnGuard Clients Summary

@ArubaNetworks |

5#ATM16

Analysis and Trending–Analysis and Trending graphs provide insight into the authentication load on the

Server and pattern of authentications - like successful versus failed authentications

@ArubaNetworks |

6#ATM16

System Monitor–System monitor provides insight into the performance metrics of the CPPM

Server in-terms of the CPU, Disk, Memory utilization and the duration of the request processing.

@ArubaNetworks |

7#ATM16

Access Tracker–Troubleshooting user authentication issues starts with Access tracker on CPPM

– this holds large chunk of information for analysis and narrowing down issues

@ArubaNetworks |

8#ATM16

Alerts Messages

@ArubaNetworks |

Error Code Error Type Cause Resolution

206 Access denied by policy User was denied access based on configured policies Verify the Enforcement Policy rules.

101 Failed to perform service classification Clearpass failed to find an appropriate service for the authentication request

Verify the incoming access request parameters against service classification rules.

201 User not found The user was not found in the authentication source Verify the authentication source about the user entry.

216 User authentication failed Incorrect username/password Request the user to double-check credentials.

225 User account disabled User account disabled in Guest DB Enable user account in Guest database.

203 Failed to contact AuthSource Authentication source did not respond in a timely manner. Verify that the authentication source (AD/LDAP/Token Server/etc) is active and can be reached by Clearpass.

9002 Request timed out Client did not respond to the authentication request. Request user to respond with a username/password/certificate credentials when prompted. Or client didn’t complete EAP transaction due to roaming etc..

9015 Client does not support configured EAP methods Client's network configuration is incorrect Request client to verify settings based on the network requirements.

 215 EAP-TLS: fatal alert by client - bad_certificate Client's network configuration is incorrect Request user to click “OK” when prompted to trust the certificate.

 215 EAP-TLS: fatal alert by server - unknown_ca Clearpass reject authentication as to client certificate validation failed. Request to verify the trust list setting and OCSP/CRL settings.

9#ATM16

Event Viewer–This page provides reports about system-level alerts and should be looked at

for any Major issues on the Server as it holds information about Critical events.

@ArubaNetworks |

10#ATM16

Audit Viewer–Use the Audit Viewer to confirm any recent changes made to server

configuration.

@ArubaNetworks |

11#ATM16

Insight – An Advanced Analytics/Reporting AppDelivers enhanced analytics, in-depth reporting, alerting and significant gains when addressing compliance and regulatory overhead. It provides the ability to track detailed authentication records, audit trails, and develop systematic reports on network-access trends.

– Insight Report: Reporting functionality in Insight helps us to monitor the pattern of authentications, context, health and proactively identify issues based on the reports. It can be used in real-time analytics, as well as the ability to look into the past to satisfy historical analysis and compliance needs.

– Templates for report: Insight includes several ready-to-use pre-configured templates that help reduce the time associated with creating custom reports. The templates guide users through the process of capturing data for a number of use cases with minimal configuration.

– In-depth Analytics. Insight uses a powerful analytics engine that mines network access logs in order to generate trending report on various parameters. Network managers can utilize these trends to get an overview of authentication and access activity, elaborate client access distribution, load-averages, and analyze authentication traffic flow through various network devices

– Alerts. Insight can generate near real-time alerts on anomalous network activity. Network managers can configure alerts based on a number of various parameters. Alerts can be delivered via SMS or e-mail notification to multiple recipients to prompt action.

We have pre-configured alerts, watchlist, folderview of alerts, ability to edit/clone alerts in 6.6.

– Insight Search: Deep dive context for user, client, ClearPass server and NAD

@ArubaNetworks |

12#ATM16

Best Practices

@ArubaNetworks |

13#ATM16

Scheduled Backup of configurationClearPass Policy Manager provides the ability to push scheduled data securely to an external server using SFTP and SCP protocols. 

@ArubaNetworks |

14#ATM16

Cluster Wide Parameters

– Auto backup settings should be set to “Off” or “Config"

– Session log details retention – 3 day default

– Known Endpoint clean up interval – Review and setup if appropriate. Depends on the nature of the deployment.

– Unknown Endpoint clean up interval – Recommend that this is enabled. We suggest 7 as a default.

– Expired Guest account clean up interval – Review and set value depending on the nature of deployment. We suggest 30 days.

– Profiled Unknown Endpoint clean up interval – We suggest 7 as the default.

– Audit records clean up interval – 7 days

– Configure Alert Notification email/SMS.

– Insight Data Retention – 30 days

@ArubaNetworks |

15#ATM16

To address issues related to AD authentication

– Authentication error MSCHAP: AD status: Named pipe disconnected

– Radius/Domain services stops frequently.

Recommendations:– Join ClearPass to domain controller which is available locally.

– Use AD password servers to configure backup DCs.

– Configure AD errors recovery action. CPPM excludes the following errors from AD errors which are used for Recovery actions.

0xC000006D - STATUS_LOGON_FAILURE, 0xC000006E - STATUS_ACCOUNT_RESTRICTION, 0xC000006F - STATUS_INVALID_LOGON_HOURS, 0xC0000071 - STATUS_PASSWORD_EXPIRED, 0xC0000072 - STATUS_ACCOUNT_DISABLED, 0xC0000064 - STATUS_NO_SUCH_USER, 0xC000006C - STATUS_PASSWORD_RESTRICTION, 0xC000006A - STATUS_WRONG_PASSWORD, 0xC0000193 - STATUS_ACCOUNT_EXPIRED, 0xC0000234 - STATUS_ACCOUNT_LOCKED_OUT, 0xC0000224 - STATUS_PASSWORD_MUST_CHANGE

@ArubaNetworks |

16#ATM16

Enabling debug and collecting logs– Enable debug for appropriate service.

– Perform test authentication/activity and collect logs.

– Collect the necessary data from server/client. (Access tracker dashboard details, client OnGuard logs ..)

– Restore the log level to default when finished troubleshooting.

@ArubaNetworks |

17#ATM16

Case Study

@ArubaNetworks |

18#ATM16

Authentication timeout issues

–We may come across situations where all the user authentications or the majority of the user authentications fail due to timeouts

– Sometimes this may be due to CPPM running out of RADIUS threads to process the requests

–The system starts working fine after either restarting the services or the server, but you would encounter issue again encounter after some time

@ArubaNetworks |

19#ATM16

Authentication timeout issuesCause

–We have observed this issues in many instances where ClearPass receives delayed response from AD, which causes the queue to pile up and reach the maximum threads allotted for the server.

–All the requests that arrive will be timed out as there are no threads to process the request against AD.

–We also need to look at the load on the ClearPass server to see if it is within the handling capacity of the particular server model (500/5k/25k) and if there is a huge increase in the load on the server at the time when the issue triggered.

@ArubaNetworks |

20#ATM16

Authentication timeout issuesTroubleshooting

– We need to check the Access Tracker for the user requests before the failure and verify the AD user lookup time is within few milliseconds and not in few seconds. We have noticed that a delayed response time of ~2 seconds from AD results in exhaustion of all the available threads which quickly causes an issue

– We can also look at an individual request/response from the AD server in the samba logs in ClearPass to confirm when the request was sent and response received.

[2015/11/16 14:22:06.202241,  3, pid=17583] winbindd/winbindd_pam.c:1834(winbindd_dual_pam_auth_crap)

  [ 2277]: pam auth crap domain: STAR user: Monica Hermosilla

[2015/11/16 14:22:17.501540,  2, pid=17583] winbindd/winbindd_pam.c:1939(winbindd_dual_pam_auth_crap)

  NTLM CRAP authentication for user [STAR]\[Monica Hermosilla] returned NT_STATUS_LOGON_FAILURE (PAM: 7)

 

@ArubaNetworks |

21#ATM16

Authentication timeout issues

@ArubaNetworks |

22#ATM16

Authentication timeout issues

@ArubaNetworks |

23#ATM16

Solving Authentication Timeout IssuesRecommendations

– AD end delays could be caused due to multiple reasons:– starting from performance issues on the server, replication issues with other domain

controllers or even due to network related issues.

– Extensive logging and capturing of packets on the AD server can help determine the amount of delay in responding to requests

– We also need to make sure there is no network lag induced if the servers are at different physical locations. It is recommended to join the ClearPass servers to a Local DC to avoid this situation.

@ArubaNetworks |

24#ATM16

Join Aruba’s Titans of Tomorrow force in the fight against network mayhem. Find out what your IT superpower is.

Share your results with friends and receive a free superpower t-shirt.

www.arubatitans.com

Thank you