8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
1/132
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
2/132
Troubleshooting Wireless LANsBRKEWN-3011
Patrick Croak
Technical Leader
CCIE Wireless #34712
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
3/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Wireless LANs
Software and Support
Troubleshooting Basics
AP Discovery/Join
WLC Config/Monitoring
Client Connectivity
Mobility
Packet Analysis
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
4/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Software and Support
Opening a TAC Service Request
Cisco Support Model
What to expect from TAC
How does escalation work?
WLC Software Trains
CCO (ED/MD/AW)
Engineering Specials
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
5/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
What should I have ready?
– Clear problem description – Always: Show run-config
– If client involved, always: debug client
– Your analysis of any data provided
– Set clear expectation of timeline and severity
Software and SupportOpening a TAC Service Request
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
6/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Software and Support
What to expect from TAC
– Configuration assistance – Problem analysis / bug isolation
– Workarounds or fixes
– Action plan to resolve SR
– Hardware replacement
– Engage BU when appropriate
Cisco Support Model - Expectations
What not to expect from
‒ Design and deployment
‒ Complete configuration
‒ Sales related informatio
‒ RF Tuning
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
7/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Software and Support
TAC Escalation Process
– Multi-Tier support resources within a technology – TAC to engage resources (TAC/BU) when appropriate
– SR ownership might not change hands
Customer Escalation Process
– Raise SR priority (S1/S2)
– Engage account team
– Your satisfaction is important to the Cisco TAC. If you have concerns aboprogress of your case, please contact your regional TAC.
Cisco Support Model - Escalation
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
8/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Software and Support
CCO - Cisco.com release
– 7.0.240.0, 7.3.112.0, 7.4.100.0, etc… – Full test cycle
– Classified as ED when posted
AssureWave
– AW is no longer tagged on CCO, but AW validation results are available http://www.cisco.com/go/assurewave
– Results available 4 weeks after CCO
MD
– MD tag represents stable releases for mass adoption
– MD tag will be considered on CCO after AW release validation, 10 weeksTAC/Escalation signoff
WLC Software Trains - CCO
http://www.cisco.com/go/assurewavehttp://www.cisco.com/go/assurewave
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
9/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Software and Support
Not all images are created equally
Diagnostic/Validation
– Debug Image
– Test Image
Special Fix
Production Ready
– Escalation Code
– Beta / Pre-Release
– CCO
WLC Software Trains - Engineering Special (ES)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
10/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Basics
Troubleshooting 101
– Clearly define the problem – Understand any possible triggers
– Know the expected behavior
– Reproducibility
Recommended Tools
– Spectrum Analyzer
– Wireless Sniffer and Wired Captures
Prob
Defin
Ques
Te
Solut
Ana
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
11/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Basics
Troubleshooting is an art with no right or wrong procedure, but bes
logical methodology. Step 1: Define the problem
– It is crucial to understand all possible details of a problem
– Knowing what is and is not working will go a long way
– With a proper understanding of the problem description you can skip ma
– Bad description: “Client slow to connect”
– Good description: “Client associations are rejected with Status17 severathey associate successfully.”
Troubleshooting 101
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
12/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Basics
Step 2: Understand any possible triggers
– If something previously worked but no longer works, there should be an trigger
– Understanding any and all configuration or environmental changes coulda trigger
Step 3: Know the expected behavior
– If you know the order of expected behavior that is failing, defining where
breaks down (Problem Description) is better than defining the end result. – Example: “One way audio between Phone A and B, because Phone A do
ARP Response for Phone B”
Troubleshooting 101
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
13/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Basics
Step 4: Reproducibility
– Any problem that has a known procedure to reproduce (or frequently ranshould be easy to diagnose
– Being able to easily validate or disprove a potential solution saves time bto quickly move on to the next theory
– If a problem is reproducible in other environments with a known proceducan facilitate internal testing and proposed fix/workaround verification
Debugs and Captures of working scenarios can help pin point whethe difference is
Troubleshooting 101
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
14/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Basics
Wireless Sniffer
– Example: Linksys USB600N with Omnipeek TAC can publish Omnipeek-RA if you have compatible HW
Windows 7 with Netmon 3.4 https://supportforums.cisco.com/docs/DOC-16398
Mac OS X 10.6+ https://supportforums.cisco.com/docs/DOC-19212
Wired Packet Capture
– Example: Wireshark Use for spanned switchports of AP/WLC or client side data
Spectrum Analyzer
– Spectrum Expert with Card or Clean-Air AP
The “Client Debug”
AP Packet Capture
Recommended Tools
https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-16398
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
15/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Wireless LANs
Software and Support
Troubleshooting Basics
AP Discovery/Join
WLC Config/Monitoring
Client Connectivity
Mobility
Packet Analysis
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
16/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Discover/Join
AP Runs Hunto Find Candid
to
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
17/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Discover/Join
AP Discovery Request sent to knownand learned WLCs
Broadcast
– Reaches WLCs with MGMT Interfacein local subnet of AP
– Use “ip helper -address ” with “ipforward-protocol udp 5246”
Dynamic – DNS: cisco-capwap-controller
– DHCP: Option 43
Configured (nvram)
– High Availability WLCsPri/Sec/Ter/Backup
– Last WLC
– All WLCs in same moblast WLC
– Manual from AP - “capcontroller ip address
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
18/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Discover/Join
WLCs send Discovery Response back to AP
– Name, Capacity, AP Count, Master?, AP-MGR, Load per AP-MGR
AP selects the single best WLC candidate from
– High Availability Config: Primary/Secondary/Tertiary/Backup
– Master Controller
– Greatest available capacity
– Ratio of total capacity to available capacity
AP sends single Join Request to best candidate
– WLC responds with Join Response
– AP joins and receives config (or downloads image if not correct)
Join Process
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
19/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Discover/Join
“Lightweight AP (LAP) Registration to a Wireless LAN Controller (W
Document ID 70333 Make sure date/time on WLC is accurate (certificates)!
NAT
Config network ap-discovery nat-ip-only
From AP
Debug ip udp
Debug capwap client events
From WLC
Debug mac addr (Radio mac if running full k9w8 imag
Debug capwap [event/error/packet] enable
Troubleshooting AP Discover/Join
http://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtmlhttp://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtmlhttp://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtmlhttp://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtml
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
20/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Wireless LANs
Software and Support
Troubleshooting Basics
AP Discovery/Join
WLC Config/Monitoring
Client Connectivity
Mobility
Packet Analysis
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
21/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
WLC Supportability
– Methods of Management
– Using the GUI
– Important Show Commands (CLI)
– Important Debugs (CLI)
– Best Practices
Supportability - WLC
AP Supportability
‒ Methods of Accessing the A
‒ Important Show Command
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
22/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Methods of Management
GUI – HTTPS (E) / HTTP (D)
CLI
– Console
– SSH (E) / Telnet (D)
SNMP – V1 (D) / V2 (E) – Change me!
– V3 (E) – Change me
Note: Management Via Wireless Clients (D)
Supportability - WLC
Default Mode
(E)=Enabled (D)=
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
23/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Using the GUI
Monitor
AP/Radio Statistics
WLC Statistics
Client Details
Trap Log
Supportability - WLC
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
24/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Using the GUI
Wireless > All APs
AP list shows AP Physical UP Time
APs are sorted by Controller Associated Time
Check bottom of AP list for any recent AP disruptions
Select AP to see Controller Associated Time (duration)
WLC Config/MonitoringSupportability - WLC
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
25/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Using the GUI
Management
SNMP Config
Logs
Tech Support
Supportability - WLC
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
26/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Important Show Commands (CLI)
Show run-config
–Must have! No exceptions!
–“show run-config commands” (like IOS show running-config)
–“show run-config no-ap” (no AP information added)
Show tech-support
CLI Tip
–Log all output
–Config Paging Disable
Supportability - WLC
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
27/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Important Debugs (CLI)
Debug client
–Client Involved? Must Have! No Exceptions
Debug capwap enable
CLI Tips
–Log all output
–Debugs are session based, they end when session ends
–“Config session timeout 60”, sets 60 minute idle timeout
–Debug disable-all (Disables all debugs)
Supportability - WLC
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
28/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Best Practices
Change default SNMP Parameters
Configure Syslog for WLC and AP
!!AP default behavior is to Broadcast syslog!!
Enable Coredump for WLC and AP Configure NTP Server for Date/Time
Supportability - WLC
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
29/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Supportability
Methods of Accessing the AP
– Console
– Telnet (D) / SSH (D)
– No GUI support
– AP Remote Commands
Enabling Telnet/SSH
– WLC CLI: config ap [telnet/ssh] enable
– WLC GUI: Wireless > All APs > Select AP > Advanced > Select [telnet/s
Default Mode
(E)=Enabled (D)=
Supportability
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
30/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Remote Commands (WLC CLI)
Debug AP enable
Enables AP Remote Debug
AP Must be associated to WLC
Redirects AP Console output to WLC session
Debug AP command “” Output is redirected to WLC session
AP runs IOS, numerous generic IOS commands available
AP SupportabilitySupportability
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
31/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Show Commands (AP CLI or WLC Remote Cmd)
Show controller Do[0/1] (or Show Tech)
Must have! Before/During/After event
Show log
WLC: show ap eventlog
Show capwap client
CLI Tips
Debug capwap console cli
Debug capwap client no-reload
AP Supportability
Supportability
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
32/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Supportability
– WLC
– AP
WLANs
RRM / Radio / RF
Wireless LAN Controller Config Analyzer (WLCCA)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
33/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
AP “Default Group” consists of all WLANs ID 1-16 and cannot be m
AP Groups must be created for WLAN ID 17+
AP Groups override the Interface configured local to the WLAN
AP Groups override default RF Profiles
WLANs – AP Groups
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
34/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/MonitoringWLANs - Tweaks
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
35/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Supportability
– WLC
– AP
WLANs
RRM / Radio / RF
Wireless LAN Controller Config Analyzer (WLCCA)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
36/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
There are generally two common scenarios or issues involving RR
APs power change frequency (too much or not at all) – Nearby APs list meets the general rule of RSSI from 3rd closest AP is be
Power Threshold
– TPC Tuning may be required
APs not changing channel
– Check if other APs are in each others neighbor list
– Already established channel plan might not change APs without just cau
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
37/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
show ap auto-rf [802.11a/b]
Load Information – Receive Utilization.. 0 % Rx load to Radio
– Transmit Utilization.. 2 % Tx load from Radio
– Channel Utilization.. 12 % % Busy
Nearby APs
– AP 00:16:9c:4b:c4:c0 slot 0.. -60 dBm on 11 (10.10.1.5)
– AP 00:26:cb:94:44:c0 slot 0.. -64 dBm on 11 (10.10.1.4)
Show AP Auto-RF (In Run-Config)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
38/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
Power Assignment Leader
Power Threshold
Consider Minimum Power Level Assignment
Radio – TPC Tuning
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
39/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
RF Profiles let you make the same TPC settings but for specific gro
Radio – TPC Tuning – RF Profiles
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
40/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
If channels change too frequently, DCA may need to be made lessrun at longer intervals
DCA Tuning
RRM / R di / RF
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
41/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
In some large
environments with new APs being deployed,STARTUP mode maybe beneficial
Previously this requireda WLC REBOOT, but
can be accomplished byRF Groupingconfiguration
DCA – STARTUP Mode
RRM / R di / RF
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
42/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RRM / Radio / RF
Clean Air can give a remote viewinto the general RF environment
around an AP
RF – Clean Air
WLC C fi /M it i
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
43/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
SE-Connect or Local Mode
Obtain Spectrum Key
Connect to Remote Sensor
Spectrum Expert with Clean Air
Spectrum Expert with Clean Air
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
44/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC C fi /M it i
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
45/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config/Monitoring
Supportability
– WLC
– AP
WLANs
RRM / Radio / RF
Wireless LAN Controller Config Analyzer (WLCCA)
WLC Config Analyzer (WLCCA)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
46/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config Analyzer (WLCCA)
Main objective: Save time while analyzing configuration files from W
Audit Checks
Support Forums DOC-1373
WLC Config Analyzer (WLCCA)
https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
47/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
WLC Config Analyzer (WLCCA)Support Forums DOC-1373
Secondary objective:
Carry out RF analysis
Troubleshooting Wireless LANs
https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
48/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Wireless LANs
Software and Support
Troubleshooting Basics AP Discovery/Join
WLC Config/Monitoring
Client Connectivity
Mobility Packet Analysis
Steps to Building an 802.11 Connection
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
49/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
1. Listen for Beacons
2. Probe Request
3. Probe Response4. Authentication Request
5. Authentication Response
6. Association Request
7. Association Response
8. (Optional: EAPOL Authentication)
9. (Optional: Encrypt Data)
10. Move User Data
State 1:
Unauthenticated,
Unassociated
State 2:
Authenticated,
Unassociated
State 3:
Authenticated,
Associated
802.11
A
Understanding the Client State
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
50/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Understanding the Client StateName Description
8021X_REQD 802.1x (L2) Authentication Pending
DHCP_REQD IP Learning State
WEBAUTH_REQD Web (L3) Authentication Pending
RUN Client Traffic Forwarding
(Cisco Controller) >show client detail 00:16:ea:b2:04:36
Client MAC Address............................... 00:16:ea:b2:04
…..
Policy Manager State............................. WEBAUTH_R
00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
The Client Debug
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
51/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
A multi-debug macro – (Cisco Controller) >debug client 00:16:EA:B2:04:36
– (Cisco Controller) >show debug
– MAC address ................................ 00:16:ea:b2:04:36
– Debug Flags Enabled:
dhcp packet enabled
dot11 mobile enabled
dot11 state enabled
dot1x events enabled
dot1x states enabledpem events enabled
pem state enabled
CCKM client debug enabled
The Client Debug
The Client Debug
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
52/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
The Client Debug• 3 Simultaneous MAC Addresses in 7.2
• Up to 10 Simultaneous MAC Addresses in 7.3 and later
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
53/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
g g
Association (Start)
L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc Tips and Tricks
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
54/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
The Client Debug Walkthrough
Association (Start)
L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc Tips and Tricks
Association
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
55/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
(Cisco Controller) >debug client 00:16:EA:B2:04:36
(Cisco Controller) >
(Cisco Controller) >
Association received from mobile on AP 00:26:cb:94:44:c0
0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:162 Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface
Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3‘
STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0
Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36
0.0.0.0 START (0) Initializing policy
0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)
0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client
0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1
apfMsAssoStateInc
apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Ass
Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
Association
Association
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
56/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association
Association received
Association Request, client did not “Roam” (Reassociate)
AP Base Radio = 00:26:cb:94:44:c0
vapId 1, site 'default-group', interface '3‘
vapId = WLAN # (Wlan 1)
site = AP Group (default-group)
Interface = Dynamic Interface name (3)
vlan 3
Vlan = Vlan # of Dynamic Interface
Association received from mobile on AP 00:26:cb:94:44:c0
0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:162
Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interfa
Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3'
Association
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
57/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association
STA - ratesMandatory Rates (>128) = (#-128)/2
Supported Rates (
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
58/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association
0.0.0.0 START
0.0.0.0 = IP we know for client (In this case nothing)
Change state to 8021X_REQDPassed association, moving client to next state: 8021X_REQD
Scheduling deletion
Session Time on WLAN (1800 seconds in this case)
0.0.0.0 START (0) Initializing policy
0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)
0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)
0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client
0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1
apfMsAssoStateInc
apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Ass
Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds
Association
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
59/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Common Assoc Response Failures:
1 – Unknown Reason – Anything not matching defined reason codes
12 – Unknown or Disabled SSID
17 – AP cannot handle any more associations (Load Balancing)
18 – Client is using a datarate that is not allowed
35 – WLAN requires the use of WMM and client does not support it
201 – Voice client attempting to connect to a non-platinum WLAN
202 – Not enough available bandwidth to handle a new voice call (CAC Reje
Association
Slot 0 = B/G(2.4) Radio
Slot 1 = A(5) Radio
Sending Assoc Response Status 0 = Success
Anything other than Status 0 is Failure
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
Association - Takeaway
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
60/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
y
Association vs. Reassociation
Debug shows AP, Slot, AP-Group, WLAN ID, Interface, Data Rates, Encryption type
Association Response
Confirms if Client is associated
Defines reason if denied
Further troubleshootingMay require Wireless Sniffer or capture at AP Switchport
If not sending Assoc Request, must know why from Client
Trying disabling WLAN features to “dumb it down”
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
61/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
g g
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks
802.1X Authentication
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
62/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
EAP-ID-Request
Rest of the EAP Conversation
Radius-Access
(Key)
EAP-Success
EAPOL-START
EAP-ID-ResponseRADIUS (EAP-ID_R
Supplicant Authenticator
The Supplicant Derives the
Session Key from User Password or
Certificate and Authentication ExchangeSe
802.1X Authentication Association + 802 1x
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
63/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association + 802.1x
Probe Request
Probe Response
Auth Request
Auth Response
Association Request
Association Response
EAP Start
EAP ID Request
EAP ID Response
EAP Method
EAP Success
EAPoL 4 way Exchange
DATA
AP W
Between 4 and
20+ frames
WPA2-AES-802.1X
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
64/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800
dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state
Sending EAP-Request/Identity to mobile 00:16:ea:b2:04:36 (EAP Id 1)
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
Username entry (cisco) created for mobileReceived Identity Response (count=1) from mobile 00:16:ea:b2:04:36
EAP State update from Connecting to Authenticating for mobile 00:16:ea:b2:04:36
dot1x - moving mobile 00:16:ea:b2:04:36 into Authenticating state
…………………..
Entering Backend Auth Req state (id=3) for mobile 00:16:ea:b2:04:36
Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3)
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)
...........................
Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 10, EAP Type 25)
Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36
Processing Access-Challenge for mobile 00:16:ea:b2:04:36Entering Backend Auth Req state (id=11) for mobile 00:16:ea:b2:04:36
Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 11)
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 11, EAP Type 25)
Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36
Processing Access-Accept for mobile 00:16:ea:b2:04:36
***OR***Processing Access-Reject for mobile 00:16:ea:b2:04:36
Common EAP Types
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
65/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
1 – Identity
2 – Notification
3 – NAK
4 – MD5
5 – OTP
6 – Generic Token
13 – EAP TLS
17 – LEAP
18 – EAP SIM
21 – EAP TTLS
25 – PEAP
43 – EAP-FAST
Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EA
WPA(2) - PSK Authentication
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
66/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Probe Request
Probe Response
Auth Request
Auth Response
Association Request
Association Response
EAPoL 4 way Exchange
DATA
AP W
WPA(2) – PSK Authentication (cont.)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
67/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
Creating a PKC PMKID Cache entry for station 00:16:ea:b2:04:36 (RSN 2)
Adding BSSID 00:26:cb:94:44:c0 to PMKID cache for station 00:16:ea:b2:04:36
New PMKID: (16)
[0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd
Initiating RSN PSK to mobile 00:16:ea:b2:04:36
dot1x - moving mobile 00:16:ea:b2:04:36 into Force Auth stateSkipping EAP-Success to mobile 00:16:ea:b2:04:36
Including PMKID in M1 (16)
[0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd
Starting key exchange to mobile 00:16:ea:b2:04:36, data packets will be dropped
Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36
state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00
Received EAPOL-Key from mobile 00:16:ea:b2:04:36
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36
Received EAPOL-key in PTK_START state (message 2) from mobile 00:16:ea:b2:04:36
Stopping retransmission timer for mobile 00:16:ea:b2:04:36Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36
state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01
Received EAPOL-Key from mobile 00:16:ea:b2:04:36
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36
Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:16:ea:b2:04:36
apfMs1xStateInc
0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state L2AUTHCOMPLETE (4)
WPA2- PSK - Failed
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
68/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Starting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be dropped
Sending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57
state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00
Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57
802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57
Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57
Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57
802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57
Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57
………………… 802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57
Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57,
retransmit count 3, mscb deauth count 3
Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57
apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1)
L2 Authentication - Takeaway
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
69/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
8021X_REQD means L2 Authentication pending
Authentication/Encryption has not be established
PSK is 802.1X, key is derived from PSK not AAA
If “Processing Access-Reject”
AAA/RADIUS Rejected the user (not the WLC)
If “Processing Access- Accept”
AAA/Radius Accepted the userM1-M4 should follow
Further Troubleshooting
Debug aaa [all/event/detail/packet] enable
Debug dot1x [aaa/packet] enable
802.1X Authentication Roaming
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
70/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Probe Request
Probe Response
Auth Request
Auth Response
Reassociation Request
Reassociation Response
EAP Start
EAP ID Request
EAP ID Response
EAP Method
EAP Success
EAPoL 4 way Exchange
DATA
AP2 W
Between 12 and20+ packets
DATA AP1
802.1X Authentication Roaming
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
71/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
802.1x + WPA2 FSR (PMKID Caching) is like PSK
Probe Request
Probe Response
Auth Request
Auth Response
Reassociation Request
Reassociation Response
EAPoL 4 way Exchange
AP2 W
DATA
AP1
6 packets
DATA
802.1X with CCKM Authentication Roaming
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
72/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
CCKM (WPA1-TKIP or WPA2-AES)
Probe Request
Probe Response
Auth Request
Auth Response
Reassociation Request
Reassociation Response
AP2 W
DATA
AP1
2 packets
DATA
Association - FSR
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
73/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Processing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36
CCKM: Mobile is using CCKM
CCKM: Processing REASSOC REQ IE
Including CCKM Response IE (length 62) in Assoc Resp to mobile
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot
FSR
CCKM - WPA
CCKM - WPA2
WPA2 PKC
WPA2 "Sticky"
OR
Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36
Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36
Received PMKID: (16)
[0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8
Found an entry in the global PMK cache for station
Computed a valid PMKID from global PMK cache for mobile
* WPA2 “Sticky” PMKID Caching is now supported in 7.2 WLC Release with limited scale
This at least allows some form of Fast Secure Roaming for “Sticky” clients (like Apple).
802.11r Roaming
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
74/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP1
Client
ProbReq
ProbResq
FT req via 802.11 auth/A ctio n
frame
FT resq v ia 802.11 auth /
Act ion frame
AssocReq wi th QOS req
AssocResp wi th QOS req
AP2
DATAtransfer
via AP1
DATA
transfe
via AP
ROAMIN
WPA2 - .11r Client (Fast Transition)
802.11r Over the Air Roaming
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
75/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP1
Client
Roaming direction
A s s o c i a t e d w i t h
o l d A P
AP2, 3, 4
8 0 2. 1
1 F T a
u t h r e q
8 0 2. 1
1 F T a
u t h r e s p
R e a s s
o c i a t i
o n R e
q
R e a s s
o c i a t i
o n R e
s p
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
76/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks
Client DHCP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
77/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state00:16:ea:b2:04:36 apfMs1xStateInc
00:16:ea:b2:04:36 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)
00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 3 apVap
00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vap00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7)
00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4755, Adding TMP rule00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
00:16:ea:b2:04:36 Stopping retransmission timer for mobile 00:16:ea:b2:04:36
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
...................
00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03)
...................
00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)
...................00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0
Client DHCP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
78/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Client is in DHCP_REQD state
Proxy Enabled:
DHCP Relay/Proxy
Between WLC and Server
Required for Internal DHCP
Proxy Disabled:
Between Client and Server
DHCP is broadcast out VLANIP helper or other means required
Client State = “DHCP_REQD
DHCP Proxy Enabled
Client DHCP Discover
Unicast to DHCP Servers
DHCP Offer from Server
DHCP ACK from Server
IP Address Learned
Client DHCP Request
DHC
Client
B
DHCP Proxy Enabled – DHCP Discover
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
79/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
32.151: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0x
32.151: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings:
dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,dhcpGateway: 0.0.0.0, dhcpRelay: 0.0.0.0 VLAN: 0
32.151: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.1
(local address 10.10.1.4, gateway 10.10.1.1, VLAN 0, port 29)
32.151: 00:16:ea:b2:04:36 DHCP transmitting DHCP DISCOVER (1)
32.151: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1
32.151: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0
32.152: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
32.152: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
32.152: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.4
32.152: 00:16:ea:b2:04:36 DHCP requested ip: 10.99.76.147
32.152: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.1 (len 346, port 29, vlan 0)
32.152: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings:
dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,
dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 0
32.152: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE
DHCP Proxy Disabled – DHCP Discover
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
80/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)
*00:16:ea:b2:04:36 DHCP processing DHCP DISCOVER (1)
*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0
*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0
*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86
*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS
Learning IP without DHCP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
81/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Client IP can be learned by ways other than DHCP
Client sends gratuitous ARP or ARP Request (Static Client)
Client sends IP packet (Orphan Packet), we learn IP
DS sends packet to client, we learn IP from DS
Seen with mobile devices that talk before validating DHCP
Up to client to realize their address is not valid for the subnet
DHCP Required on WLAN for preventing this
*Orphan Packet from 10.99.76.147 on mobile
*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
*Installing Orphan Pkt IP address 10.99.76.147 for station
*10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
Client DHCP - Takeaway
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
82/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
DHCP_REQD means Learning IP State
Only “Required” if enabled on the WLAN
If Proxy is enabledConfirm DHCP Server on Interface (or Wlan) is correct
DHCP Server may not respond to WLC Proxy (Firewalls?)
If Proxy is disabled, DHCP is similar to wired client
Further Troubleshooting
Check DHCP Server for what it believes is happeningIf WLC does not show a BOOTREQUEST, confirm the client request arrive
and leaves in the configured way
If still believed to be on WLC: debug dhcp message enable
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
83/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks
Webauth*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
84/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xe
……………………………...
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to WEB
last state WEBAUTH_REQD (8)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP r*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL I
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile
*pemReceiveTask: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 2, dtlFlags 0x0*pemReceiveTask: 00:16:ea:b2:04:36 Sent an XID frame
*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile
*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile
*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile………………………………
*emWeb: 00:16:ea:b2:04:36 Username entry (cisco) created for mobile*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) last state WEBAU
*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last st*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile
*emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063
*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, Toke
3, IPv6 intf id = 8
*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)
*pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1,
Webauth RedirectWebauth
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
85/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Client in WEBAUTH_REQD state
ARP and DNS must be functional
Client attempts to browse internet
WLC “Hijacks” the handshake
Client redirects to Virtual Interface
Certificate negotiation if applicable
Webauth page is displayed
Client authenticates
Client State
“WEBAUTH_RE
ARP and DNS Fu
3-Way Handshake
HTTP GET
200 Respons
3-Way Handsh
HTTP(S) GE
Successful Authen
Client State = “R
Webauth Page Dis
Confirm ARP and DNS Function
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
86/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AR
Capture from Wireless Adapter
Webauth Redirect
WLC Re
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
87/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Webauth Redirect WLC ReS
WLC Re
S
Address for Cl
Redirect to (V
IP/Name
Redir
Interfac
Clie
If WEBAUTH REQD, then not authenticated
Webauth - Takeaway
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
88/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
If WEBAUTH_REQD, then not authenticated
Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6*(7.0 and ear
If not redirected, can client browse to virtual IP?
Cert issue? Consider disabling HTTPS for HTTP webauth
Most common scenario involves ARP/DNS failure
Must confirm that client actually sends TCP SYN (http) to IP
If proven that TCP SYN is sent and WLC does not SYN ACK, then
be a WLC side problemdebug client debug webauth enable
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
89/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks
Run State
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
90/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
RUN State is the Client Traffic Forwarding State
Client is Connected and should be functional
10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 5273
10.10.3.82 Added NPU entry of type 1, dtlFlags 0x0
OR
10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14)
10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)
Session Timeout is 1800 - starting session timer for the mobile
10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063
10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
91/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks
Deauthenticated Client Idle Timeout
O ft t ffi i d f Cli t t AP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
92/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Occurs after no traffic received from Client at AP
Default Duration is 300 seconds
Session Timeout
Occurs at scheduled duration (default 1800 seconds)
Received Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:26:cb:94:44:c0 from Associated to Disassociated
Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
Deauthenticated Client WLAN Change
f
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
93/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Modifying a WLAN in anyway Disables and Re-enables WLAN
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile
00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
Manual Deauth
From GUI: Remove Client
From CLI: config client deauthenticate
Deauthenticated Client
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
94/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Retransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb decount 0
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)
Authentication Timeout
Auth or Key Exchange max-retransmissions reached
Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0)
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile
00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)
AP Radio Reset (Power/Channel)
AP disasassociates clients but WLC does not delete entry
Deauthentication - Takeaway
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
95/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Client can be removed for numerous reasons
WLAN change, AP change, configured interval
Start with Client Debug to see if there is a reason for a client’s dea
Further Troubleshooting
Client debug should give some indication of what kind of deauth is happen
Packet capture or client logs may be require to see exact reason
The Client Debug - Walkthrough
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
96/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks
Tips and Tricks
C ll li d b f d d d i
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
97/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Collect a client debug for an extended duration
Several roams, deauths, failures, etc…
Use an enhanced text editor with filter or “find all” I use Notepad++
Find All
“Association Received” (will also pull reassociations)
“Assoc Resp”
“Access-Reject”
“timeoutEvt”
Tips and Tricks
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
98/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Tips and Tricks
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
99/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Troubleshooting Wireless LANs
S ft d S t
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
100/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Software and Support
Troubleshooting Basics
AP Discovery/Join
WLC Config/Monitoring
Client Connectivity
Mobility
Packet Analysis
Mobility—Intra-ControllerClient Roams Between Two APs on the Same Controller
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
101/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Mobility—Inter-Controller (Layer 2)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
102/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Mobility—Layer 3
Layer 3 roaming (a.k.a. anchor/foreign)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
103/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
New WLC does not have an interface on the subnet the client is on
New WLC will tell the old WLC to forward all client traffic to the new WLC
Asymmetric traffic pathestablished
(deprecated)
Symmetric traffic path
–
Mobility— L2 Inter WLC
OldController
Client
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
104/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
New
Controller
Controller
3. mmMobileAnnounce
1.Association Req.
4. mmMobileHandoff
2.Association Resp.
Local
DATA
DATA
Debug Client
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
105/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
MobileAnnounce
MobileHandoff
Mobility— L3 Inter WLC
OldController
Client
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
106/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
New
Controller
3. mmMobileAnnounce
1.Association Req.
4. mmMobileHandoff
2.Association Resp.
Foreign
DATA
DATA
Anchor
(EOIP) DATA
Mobility— L3 Inter WLC Debug Client
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
107/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
MobileAnnounce
MobileHandoff
Mobility— L3 Inter WLC Debug Client
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
108/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Anchor
Mobility Group vs. Mobility Domain
Mobility Group - WLCs with the same group name
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
109/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Mobility Group WLCs with the same group name
L2/L3 Handoff
Auto Anchoring
Fast Secure Roaming
APs get all of these as a Discover candidate
Mobility Domain - WLCs in the mobility list
L2/L3 Handoff
Auto Anchoring
Sent between all WLCs, by member with lowest MAC
– Control Path = UDP 16666 (30 Seconds)
Mobility Data/Control Path
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
110/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
– Data Path = EoIP Protocol 97 (10 Seconds)
– debug mobility keep-alive enable
Troubleshooting Wireless LANs
Software and Support
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
111/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
pp
Troubleshooting Basics
AP Discovery/Join
WLC Config/Monitoring
Client Connectivity
Mobility
Packet Analysis
Wireshark Tutorial
Default Wireshark view might look like this:
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
112/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Wireshark Tutorial
Newer versions of Wireshark have a feature for “Apply as Column”
This will take any decodable parameter and make a column
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
113/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
y p
Wireshark Tutorial Within seconds your wireshark can also have:
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
114/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Wireshark Tutorial
Filtering data is just as easy
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
115/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Wireshark Tutorial - CAPWAP
User data is encapsulated in CAPWAP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
116/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Wireshark Tutorial Wireshark can also de-encapsulate CAPWAP DATA
Edit > Preference > Protocols > CAPWAP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
117/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Wireshark Tutorial
With CAPWAP de-encapsulated you can see all the packets to/from(b t AP d WLC)
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
118/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
(between AP and WLC)
Sniffer Mode AP
Select channel to Sniff
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
119/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Select destination for traffic
Sniffer Mode AP
Omnipeek has a Remote Adapter to capture this data
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
120/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Wireshark, just capture network adapter
NOTE: Wireshark does not open the port UDP 5000PC will send ICMP Unreachables
Sniffer Mode AP
With wireshark, filter !icmp.type == 3
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
121/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Data (UDP 5000) still not intelligible yet
– Decode as Airopeek (Peekremote in wireshark 1.8+)
Sniffer Mode AP
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
122/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
AP Packet Dump
In 7.3 WLC release, we added an AP packet dump feature that canpackets from a wireless client at the AP radio.
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
123/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
p
Much easier than performing an Over-The-Air capture, can be perf
remote locations The APs will send the packet dump to the configured FTP server
AP Packet Dump – FTP Server Required
Feature requires use of a standard FTP server running on a netwoworkstation, or laptop i.e. IIS, Filezilla, WS FTP, 3CD, etc.
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
124/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
, p p , , , ,
FTP server needs to be accessible by the APs capturing packets n
controller Multiple simultaneous file upload connections will be initiated to the
—One for the AP designated in the start command
—One for each AP that is an RF neighbor of the AP desigthe start command – on th e same contro l ler only
File name format example: 3602-15508-223042013 _ 160038.pcap AP Name
Controller NameDate ddmmyyyy
Time hhmmsec
AP Packet Dump Commands
config ap packet-dump ftp serverip path usernampassword
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
125/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
(Cisco Controller) >show ap packet-dump status
Packet Capture Status............................ StoppedFTP Server IP Address............................ 172.16.0.11FTP Server Path.................................. \FTP Server Username.............................. ciscoapFTP Server Password.............................. ********Buffer Size for Capture.......................... 4096 KBPacket Capture Time.............................. 10 MinutesPacket Truncate Length........................... Unspecified
Packet Capture Classifier........................ 802.11 Data
AP Packet Dump Filters
• First define packets to be captured by enabling specific classifiers CLI
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
126/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
— config ap packet-dump classifier enable/disable
— Only the following pre defined classifiers are available• arp• broadcast• control• data• dot1x• iapp• Ip• management• multicast
• tcp• udp
• Classifiers are enabled one at a time - more than one classifier can be time
Starting the Packet Dump• Start the dump process from the controller CLI using
– config ap packet-dump start
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
127/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
• Packet dump ends either when the capture timer expires or the promanually stopped from the controller CLI using
–config ap packet-dump stop
(Cisco Controller) >config ap packet-dump start 00:24:d7:45:4e:6c 3602-
Client Mac Address............................... 00:24:d7:45:4e:6c
FTP Server IP.................................... 172.16.0.11
FTP Server Path.................................. \
FTP Server Username.............................. ciscoap
Buffer Size for Capture.......................... 4096 KB
Packet Capture Time.............................. 10 Minutes
Packet Truncate Length........................... Unspecified
Packet Capture Classifier........................ 802.11 Data
Are you sure you want to start capture ? (y/N)
Files are not created until
you answer yes here
AP Packet Dump - dot1xThe 802.11 authentication & asso
The dot1x process begi
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
128/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
The dot1x process begi
The dot1x proc
The remaining encrypted
packets provide little useful
information
AP Packet Dump – Open/Webauth
The 802.11 authentication & asso
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
129/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
The DHCP
Process
Details
Available
Summary - Key Takeaways
Accurate Problem Description is crucial
Understand the flow for a successful client connection, determine w
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
130/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
,failing
Know the tools that are available – Debugs, show commands
– Packet captures – sniffer mode, AP packet dump
– WLCCA for configuration analysis
A few commands can go a long way
– show run-config
– debug client xx:xx:xx:xx:xx:xx
Complete Your Online Session Evaluation
Give us your feedback andyou could win fabulous prizes.Wi d d il
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
131/132
© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public
Maximize your Cisco Live exp
free Cisco Live 365 account. DPDFs, view sessions on-dema
live activities throughout the y
Cisco Live 365 button in your
log in.
Winners announced daily.
Receive 20 Cisco Daily Challengepoints for each session evaluationyou complete.
Complete your session evaluationonline now through either the mobileapp or internet kiosk stations.
8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours
132/132
Top Related