- 1.Control and Forwarding plane
- 100-Mbpsfxp1Ethernet link is used between RE and PFE
- For M320 case, 100-Mbps Ethernet switch is being used to
provide a dedicated link to each FPC. For RE, these links are
presented atbcm0
- Fxp0: management interface
- Fxp2: communication between Primary RE and backup RE
- 3)Forwarding table (FT) can hold over 800,000 routes.
2. Difference between M7i and M10i
- Redundant RE: M10i support, not M7i
- Built-in Adaptive Service: M7i. M10i needs an external AS
PIC.
3. System storage
- Compact Flash(ad0) : built-in at the board.
4. JUNOS CLI basics
- Space bar to complete a command
- Command:Help topic for general concepts
- Command : help reference for configuration syntax
- Rebooting system:request system reboot
- Shut down system:request system halt
- Log and Trace files are located at /var/log
- Command:Show log | messages |file-name
- At more prompt, use forward slash(/) to search or use h to get
a context help screen
- - show log messages | match so-0/3/1 | match TRAP--- AND
--
- - show log messages | mach fpc | sfm | kernel--- OR ---
- Monitor log/trace in real time:monitor startfile-name| match
fail
- Stop monitoring in real time:monitor stop
- Enable/disable real-time output to screen: Esc-Q
- Stop traceing operation:delete flag open
- Truncate(clear) log/trace files:clear logfile-name
- Delete log/trace files:file deletefile-name
5. JUNOS CLI basics
- Entering configuration:Typeconfigureoredit
- Exclusive configuration (configure exclusive) and Private
configuration (configure private??)
- Moving within the configuration hierarchy: edit (equivalent to
cd), up, top, exit (to previous location in the hierarchy)
- Show command at configuration mode vs. show command at
operational-mode
- Relative configuration commands Starting with JUNOS5.3 :
top
- - top show system login(show system login no matter where you
are.Examples:
- - top edit protocols ospf( to enter protocols ospf no matter
where you are)
- Viewing configuration in operational mode:show configuration
< configuration path>
- View configuration with set: show xxx | display set
- Viewing candidate configuration: show chassis alarm,show (at
the current sub-hierarchy)
6. JUNOS CLI basics
- Change the candidate configuration. Examples:
- - set alarm sonet lol red
- Display difference between the candidate and active
configurations:
- At the current statement-path, show | compare
- Viewing difference in files. Example:
- - file showfilename1| compare filefilename2
- - show configuarion | compare rollback number
- Removing statements:delete
- Delete the statements and all its subordinate statements and
identifieres.
- Wildcard delete. Example:wildcard delte interfaces fe-*
- Ignore portion of the configuration
hierarchy:deactivate/activate
- Disable an interface: set disable interface
- Delete and disabled interface: delete interface disable
7. JUNOS CLI basics
- commit ----- candidate file is checked, actived and marked as
the current
- operational sofware configuration file.
- commit check ----- only validate a candidate configuration
without
- rollback n -------- recover the previous configuration. And
then commit
- rollback 0is current configuration
- First 3 roll back (1-3) are stored in solid-state flash
disk
- /config/juniper.conf.n (n=1-3)
- rest roll back (4-49) are stored in hard
disk/var/db/config
- commit confirmedtime-out---- temporarily activate a
configuration (default is 10 minutes). If the final commit is not
executed, the system will performs a rollback 1, commit
commands.
- commit synchronize ---- after committed on the master RE
internally
- copied and committed on the backup RE automatically.
- commit attime----- commit at some time
- clear system commit ---- cancel a pending commit
8. JUNOS CLI basics
- save terminal-- for copy and paste into other others
- show | display set create configuration for simplifying
configuration editing.
- Loading configuration files ( load and then commit)
- load override filenameoverride the currentconfig with the
loaded one. Do it at the root of the configuration hierarchy.
- load mergefilename - combine the new and old
- load merge terminal(then copy/paste hierarchical
configuration)
- load replacefilename statements with replace tag will replace
the statements with the same name
- load relativeload at where it is current at the configuration
hierarchy.
9. Junos CLI Basics
- Only save the configuration under certain hierarchy. To save
the whole configure, issue this command at the top of the
hierarchy.
- Display the contents of the file you saved
- To load a configuration after clear the current
configuration
- To recover a mistake made previously after committing.
10. Junos CLI Basics
- Show log interactive-commands | match restart
- Use sysctl a to display kernel parameters.
- sysctl a | grep icmp(under shell prompt)
- show chassis 0 pic slot 1 information.
- Show chassis pic fpc-slot 0 pic-slot 1
- Request chassis cfeb master switch
- Request chassis routing-engine master switch
11. Junos CLI Basics
- Find out who logins the system and kick out some particular
users.
- lab@santro-re0> help syslog ACCT_ACCOUNTING_FERROR
- Name:ACCT_ACCOUNTING_FERROR
- Message:Unexpected error from file
- Help:Error occurred during file processing
- Description:An error prevented the accounting statistics
process from processing the indicated file.
- Type:Error: An error occurred
12. Junos CLI Basic
- show configuration with inheritance
- show configuration interfaces ge-4/3/3 |
displayinheritance
13. Syslog
- set system syslog file messages any notice
14. Hardware troubleshooting process
- Show chassis craft-interface
- Monitor start [message | chassid]
15. Display PIC status
- Show chassis pic fpc-slot 0 pic-slot 1
- lab@santro-re0> show chassis pic fpc-slot 0 pic-slot 1
- FPC slot 0, PIC slot 1 information:
- Type10x 1GE(LAN), 1000 BASE
- Uptime1 day, 22 hours, 25 minutes, 17 seconds
- PortCable typetypeXcvr vendorpart numberWavelength
- 0GIGE 1000SXSMFINISAR CORP.FTRJ8519P1BNL-J2850 nm
16. Boot image
- If you need to reboot from PCMCIA card, you need to copy a
special image called jinstall-mediaxxxx .
- Hit space when the system is rebooting until it goes to either
boot: or OK prompt.
- If you get boot: prompt, the loader is not run yet. You need to
do this:
- Change a boot device at OK prompt
- Ok nextboot compact-flash
17. Interfaces
- Disable(admin down) an interface
18. RE overview (Q: how to find out RE Platform compatibility
list?)
- Primary coopy of JUNOS resides on the flash memory.Use this
command to create a backup copy:
- RE has different versions: RE-333, RE-400, RE-600, RE-1600.
Each RE is supported by certain platforms.
- RE uses Intel processor from P III to P IV.
- Use this command to find out what RE is being used:show chassis
hardware.
- Hard disk monitoring: Self-Monitoring Analysis and Reporting
Technology System(SMART). From 5.5, SMART is enabled by default. To
disable:
- set system processes disk-monitoring disable
- Configuration file compression: default starting Release 7.0
(maybe). To enable:
- set system compress-configuration-file
- RE5(RE-400): only supported in M7i and M10i
- RE4(RE-600): All M and T series. Except M7i/M10i/M320. The only
RE to have flash memory upgrade
- RE3 (RE-333): M5/10/20/40/40e, and M160
- RE-1600: M320 and T320/T640. Using Broadcom chipset for
Ethernet connectivity to PFE.
- While used on M320, the GE link is supported as bcm0. While on
T-series, 100- Mbps is supported(???)
19. PFE overview on M-series
- Different names but referring to the route lookup module:
- M40 System Control Board (SCB)
- M20 System Switch Board (SSB)
- M5/10 FPC and SCB are combined into a single boardcalled the
Forwarding Engine Board (FEB)
- M7i/10i Compact FEB (CFEB)
- M40e and M160 Switching and Forwarding Module (SFM).4 SFM on
M160, each one provides 25% of lookupcapability.2 SFM on M40e, only
one can be active.
- Special stuff on M40e and M160 platform:
- MCS card(Miscellaneous Control Subsystem): provide control and
monitoring functions for the various components in the chassis
- PCG(PFE clock generation): 125-MHZ signal. Redundant PCGS
20. PFE on T-series and M320
- M320 is different than T and M-series. It is a combio of two
using I and J chips.
- T640 PFC2 has single PFE, PFC3 has two PFE
- T-Series nonblocking cross-bar switch fabric Switch Interface
Boards(SIBs).
- T320: 3 SIBs with 2 are active. SIB 1 and 2 are active, SIB0 is
standby. SIB0 has only one high-speed line (HSL) connected to FPC.
SIB1 and SIB2 has 2 HSL. So when SIB0 becomes active, system
performance is degraded.
- T640: 5 switch fabric cards or SIBs, 4 are active, 1 standby.
Something like Ciscos GSR.
- M320 FPC1: use single I chip
- M320 FPC2: dual I chip, thus two PFE
- M320 FPC3: dual J chip, thus two PFE
21. Physical Interface Cards (PIC)
- IP service PIC is to hardware assist complex packet processing
and has no physical ports.
- 1)Tunnel service PIC for IP-IP, GRE tunnel and PIM-SM
tunnel.
- 2)Multlink PIC:Multilink Point-to-Point (MLPPP) and Multilink
Frame Relay (MLFR, FRF 1.5)
- Hot-Pluggable except M20 and M40 which need to remove FPC.
- Take PIC offline before physically removing it. Otherwise would
cause system damage or PFE reset.
- Packet loss is expected on M-serials except M320 because of FPC
reset.
22. Flexible PIC Concentrator (FPC)
- Support 1 to 4 PICs.M160 OC-192 has an FPC support only one
PIC.
- Each FPC on M-serial pooled to create shared memory switch
fabric. So hot-swap FPC cause system to repartition the shared
memory pool; 200 ms packet loss.
- FPC is hot-swappable in all platforms except M5 and M10 which
is using FEB. However M7i and M10i are OK even using CFEB.
- Build-in FPC at some high-speed quad-wide PICs such as
OC-48c/STM-16 for M20/40. OC-192c/STM-64 SONET/SDH on M160.
- New FPC to support reuse of old PICs:
- M160 FPC1:intend to reuse M20/40 PIC
- M160 FPC2: design to support M160 only PIC, such as OC-48c
- FPC3: support native T-series PICs.
- T640 only support FPC2 and FPC3.
- set chassis fpc power off
23. M-series System Board
- M40 System Control Board (SCB)
- M20 System Switch Board (SSB)
- M5/10 FPC and SCB are combined into a single boardcalled the
Forwarding Engine Board (FEB)
- M7i/10i Compact FEB (CFEB)
- M40e and M160 Switching and Forwarding Module (SFM). 4 SFM on
M160, each one provides 25% of lookupcapability.2 SFM on M40e, only
one can be active.
- - 2 ndgeneration Internet Processor II ASIC (not on M5/10 and
M7i/10i)
- - support 840K routing entries, double from old board
420K.
- - Double on-chip memory to 16MB on IP II
- - CPU memory 128 M for M40, 256M for M20, M40e and M160.
- - Increased CPU speed to 256 MHZ.
- - First shipped with JUNOS 5.5 Sep 2002.
24. IP II ASIC
- Performance: 40 Mpps, 40 byte with 80K prefixes at routing
table.
- Packet processing features:
- Filtering, sampling, logging, counting, load balancing
- All M-series have enhanced S-board which as IP II ASIC. M5/10
doesnt have enhanced S-board.
- T-series might contain as many as 16 IP II ASIC. Each FPC has
one or two PFE which contains its own IP II ASIC.
25. Craft Interface
- Collection of mechanisms on M-series and T-series
- View System status messages
- On the front of the chassis
- FPC/PIC online/offline buttons.
- LCD screen provide status reporting for the entire system.
- What alternatives on other platforms?
- M7i: FIC (Fixed Interface Card)provide PIC offline/online
buttons
- M10i: HCM (High-Availability Chassis Manager) Card provide PIC
offline/online bottons.
26. Password recovery
- Power cycle the RE and watch it booting up
- Enter a space character at the boot loader quick help manue to
get a command prompt (dont enter space too quickly)
- When system boots up, answer recovery to recover password
- Follow the on-screen steps to change password
27. Coredump analysis using syslog message
- Step 1: Get the stack trace from syslog messages
- lab@hissy> show log messages | find "machine check"
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 machine check caused by
error on the PC
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detect register 1:
0x08, 2: 0x00
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error ack count = 0
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error address:
0x08004014
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI bus error status
register: 0x02
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 was the PCI master
- Dec 5 01:51:17 hissy tnp_sfm_3 C/BE bits: I/O read
[0b0010]
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detection reg1: PCI
cycle
- Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI status reg: parity
error
- Dec 5 01:51:17 hissy tnp_sfm_3 ^B
- Dec 5 01:51:17 hissy tnp_sfm_3 last message repeated 7
times
- Dec 5 01:51:17 hissy tnp_sfm_3 Registers:
- Dec 5 01:51:17 hissy tnp_sfm_3 R00: 0x000e8c4c R01: 0x0775dad4
R02: 0x0000334
- Dec 5 01:51:17 hissy tnp_sfm_3 R04: 0x0775dae0 R05: 0x00142e34
R06: 0x06006b3
- Dec 5 01:51:17 hissy tnp_sfm_3 R08: 0x00142e4c R09: 0x88000000
R10: 0x0000000
- Dec 5 01:51:17 hissy tnp_sfm_3 R12: 0x00100004 R13: 0x000cc411
R14: 0x0000c43
28. Coredump analysis using syslog message
- Dec 5 01:51:17 hissy tnp_sfm_3 R16: 0x00000000 R17: 0x00041410
R18: 0x0004c42
- Dec 5 01:51:17 hissy tnp_sfm_3 R20: 0x0002c490 R21: 0x00110000
R22: 0x0000000
- Juniper Confidential. For Internal use only.
- Dec 5 01:51:17 hissy tnp_sfm_3 R24: 0x00000001 R25: 0x00000000
R26: 0x0775db1
- Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback:
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc =
0x000e8c4c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc =
0x0005cd9c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc =
0x00108914
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc =
0x00108888
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc =
0x000eec84
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc =
0x00037e78
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc =
0x000380f8
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc =
0x000eeadc
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc =
0x000eefd0
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc =
0x000f0184
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc =
0x000b28cc
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc =
0x000b29f4
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc =
0x000b2a8c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc =
0x000b2c80
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc =
0x000b2d5c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc =
0x0002665c
29. Coredump analysis using syslog message
- What do I want? I will copy the following into a file called
stack
- Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback:
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc =
0x000e8c4c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc =
0x0005cd9c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc =
0x00108914
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc =
0x00108888
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc =
0x000eec84
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc =
0x00037e78
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc =
0x000380f8
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc =
0x000eeadc
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc =
0x000eefd0
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc =
0x000f0184
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc =
0x000b28cc
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc =
0x000b29f4
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc =
0x000b2a8c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc =
0x000b2c80
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc =
0x000b2d5c
- Dec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc =
0x0002665c
30. Coredump analysis using syslog message
- Step2: Find out which version and build of the image.
- So it is on M160, 4.4B3.2 and build 4.4-20010408-b20191
- lab@hissy> show version brief
- JUNOS base [4.4B3.2] (Export restricted edition)
- JUNOS Kernel Software Suite [4.4-20010408-b20191]
- JUNOS Routing Software Suite [4.4-20010408-b20191]
- JUNOS Packet Forwarding Engine Support
[4.4-20010408-b20191]
- JUNOS Online Documentation Files [4.4-20010408-b20191]
31. Coredump analysis using syslog message
- Step 3:Find out which symbol file to use.
- debug package for the crashing code if the crash is in the
kernel or routing, or the normal
- package for the PFE.The perl script jemsym can be used to
decode
- 20010201-0805@ 20010217-0805@ 20010305-0805@ 20010320-0910@
20010405-0810@
- 20010202-0805@ 20010218-0805@ 20010306-0805@ 20010321-0910@
20010406-0810@
- older dailies for released versions;
- single% cd /volume/ftp/private/unregressed/
- 3.4/ 4.0/ 4.1/ 4.2/ 4.3/ 4.4/ 5.0/
- single% cd /volume/ftp/private/junos/
- 4.0B1/ 4.0R5/ 4.1R4/ 4.3B1.2/ 4.4B2.1/
- 4.0B2/ 4.1B1.1/ 4.2B1.1/ 4.3B2.1/ 4.4B3.2/
32. Coredump analysis using syslog message
- single% cp
/volume/build/20010408-0810/jpfe-4.4-20010408-b20191-debug.tgz
.
- single% tar zxfv jpfe-4.4-20010408-b20191-debug.tgz
fpc.sym - M20/M40 fpc stack traces fpc160.sym -- M160 fpc stack
traces sbr.sym -- M5/M10 stack traces scb.sym -- M40/M20 S-Board
traces sfm.sym --M160 SFM traces. 33. Coredump analysis using
syslog message
- $Id: jemsym,v 1.7 1998/04/21 01:15:33 jim Exp $
- This file takes a Juniper panic stack trace and turns it
- # into a user-readable output from the symbol table file
- # for the running micro-kernel.
- Juniper Confidential. For Internal use only.
- By default, gmake produces a symbol table file for each
- # target, and then you run the text of the panic stack
trace,
- # perhaps saved to a temporary file, as follows:
- cat temp-backtrace_file | jemsym target.sym
34. Coredump analysis using syslog message
- Step 4: Do the stack trace
- single% cat stack | ~dbovis/bin/jemsym
usr/share/pfe/sfm.sym
- 0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x120
- 0x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x58
- 0x00108914 bchip_write_sram_opaque (0x00108898) +0x7c
- 0x00108888 bchip_write_sram_hton (0x00108878) +0x10
- 0x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x20
- 0x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c
- 0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc
- 0x000eeadc bchip_mem_test (0x000eea08) +0xd4
- 0x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa0
- 0x000f0184 bchip_probe_diag (0x000f00fc) +0x88
- 0x000b28cc cm_probe_slot (0x000b284c) +0x80
- 0x000b29f4 cm_probe_slots (0x000b297c) +0x78
- 0x000b2a8c cm_probe_chassis (0x000b2a64) +0x28
- 0x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe8
- 0x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb4
- 0x0002665c thread_suicide (0x0002665c) +0x0
35. Coredump analysis using syslog message
- Step 4: Do the stack trace
- single% cat stack | ~dbovis/bin/jemsym
usr/share/pfe/sfm.sym
- 0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x120
- 0x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x58
- 0x00108914 bchip_write_sram_opaque (0x00108898) +0x7c
- 0x00108888 bchip_write_sram_hton (0x00108878) +0x10
- 0x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x20
- 0x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c
- 0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc
- 0x000eeadc bchip_mem_test (0x000eea08) +0xd4
- 0x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa0
- 0x000f0184 bchip_probe_diag (0x000f00fc) +0x88
- 0x000b28cc cm_probe_slot (0x000b284c) +0x80
- 0x000b29f4 cm_probe_slots (0x000b297c) +0x78
- 0x000b2a8c cm_probe_chassis (0x000b2a64) +0x28
- 0x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe8
- 0x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb4
- 0x0002665c thread_suicide (0x0002665c) +0x0
36. Coredump analysis using core files
- Where to get coredump files?
- 1) Coredump files are stored at:
/volume/ftp/pub/incomfing//
- /volume/ftp/pub/incoming/2008-0104-0511
- 2) For some freaking .tgz file, you need to dothis
- gunzip < cosd.core-tarball.0.tgz.2 | tar -xvf -
- http://jtac-tools.juniper.net/crashdecode/coredump.html
- Step 1 : Using Jdebug to find out the stack traces.
- jdebug='/volume/buildtools/bin/jdebug
- /volume/buildtools/bin/jdebug
- Examples: The core file is saved
at/volume/ftp/pub/incoming/2008-0104-0511/ core-SSB0.core.0
- Step 2 : Use query-pr to find out the possible PRs based on the
stack trace.
- query-pr -m "thread_debug" -m "sched_suspend_thread"
summary
37. Coredump analysis using core (continued)
- -bash-2.05b$ /volume/buildtools/bin/jdebug
core-SSB0.core.0
- GNU gdb 6.5 juniper_2006a_411
- Copyright (C) 2006 Free Software Foundation, Inc.
- GDB is free software, covered by the GNU General Public
License, and you are
- welcome to change it and/or distribute copies of it under
certain conditions.
- Type "show copying" to see the conditions.
- There is absolutely no warranty for GDB.Type "show warranty"
for details.
- This GDB was configured as "--host=i386-unknown-freebsd4.11
--target=powerpc-juniper-eabi".
- format_string=0x25f204 "CCHIP: Too many SRAM parity errors;
restart required ")
- at ../ukern/cpu-ppc/ppc603e_panic.c:63
- format_string=0x25f204 "CCHIP: Too many SRAM parity errors;
restart required ")
- at ../ukern/cpu-ppc/ppc603e_panic.c:63
- #10x0018bf7c in cchip_error_hardware (C=0x35,
hwerror=402653184)
- at ../common/drivers/cchip/cchip_int.c:238
- #20x0018c158 in cchip_error_scan () at
../common/drivers/cchip/cchip_int.c:352
- #30x0006baec in pfe_error_scan (info=0x0) at
../common/toolkits/pfe/pfe_scb.c:172
- #40x000da8c8 in cm_handle_pfe_error (rate_limit=FALSE)
- at ../common/applications/cm/cm_pfe_restart.c:1463
- #50x000dabc0 in cm_restart_handle_timer_event (timer=0x35)
- at ../common/applications/cm/cm_pfe_restart.c:1652
- #60x000daff0 in cm_restart_event_loop () at
../common/applications/cm/cm_pfe_restart.c:1898
- #70x00026fa0 in thread_wake (thread=0x210000) at
../ukern/common/thread.c:572
38. Coredump analysis core file from special image
- Step 1: to find out the image path using what on image or core
file.
- -bash-2.05b$ what core-SSB0[1].core.3 core-SSB0[1].core.3:
- scb release 8.2I20071212_2313_pgoyette built by pgoyette on
2007-12-12 23:14:53 UTC
-
jtac-bbuild01.juniper.net:/b/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb
- -bash-2.05b$ cd /volume/nfsbuild40
- jcanopgoyetteramanathansdoshiyuris
-
/volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb
- Step 2: Find out the *.elf file. In the above case, it is
scb.elf under the above path.
39. Coredump analysis core file from special image
- Soemtimes it take more trouble to untar the compressed jpfe
file to get the elf file.
- lab@iggy> show version brief | grep packet
- JUNOS Packet Forwarding Engine Support
[4.0-20000608-s22432]
- ( From above number I dont know where to get the jpfe file
)
- single% tar zxfv jpfe-4.0-20000608-regressed-debug.tgz
fpc.sym M20/M40 fpc stack traces fpc160.sym M160 fpc stack
traces sbr.sym M5/M10 stack traces scb.sym M40/M20 S-Board traces
sfm.sym M160 SFM traces. 40. Coredump analysis core file from
special image
- -bash-2.05b$ /volume/cross/cygnus-i386-ppc/bin/gdb-core.ppc -nw
/volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb/scb.elf
core-SSB0[1].core.3
- Copyright 1997 Free Software Foundation, Inc.
- GDB is free software, covered by the GNU General Public
License, and you are
- This GDB was configured as "--host=i386-unknown-freebsd2.2.5
--target=powerpc-eabi"...
- #0topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE)
- at ../common/toolkits/topo/topo.c:428
- ../common/toolkits/topo/topo.c:428: No such file or
directory.
- (gdb) bt
-----------------------------------------------------------
- #0topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE)
- at ../common/toolkits/topo/topo.c:428
- #10x155a84 in nh_indirect_add_sub (nh=0x2163a3c,
unilist=0x0,
- indirect_elementpp=0x2163a98)
- at ../common/applications/nh/nh_indirect.c:193
- #20x155a84 in nh_indirect_add_sub (nh=0x2163a3c,
unilist=0x0,
- indirect_elementpp=0x2163a98)
- at ../common/applications/nh/nh_indirect.c:193
- at ../common/applications/pfeman/pfeman_rt.c:413
- #11 0x276cc in thread_suicide () at
../ukern/common/thread.c:951
41. Coredump analysis Kernel core ofspecial image
- Find out where is the symbol file by using what.
- Ex:/volume/nfsbuild40/pgoyette/VZ-8.2I20071212_2313/ ship /
jkernel-8.2I20080311_1541_jtac-builder-debug.tgz
- copy the jkernel file to your home directory and unzip it.
- Ex: gunzip <
jkernel-8.2I20080311_1541_jtac-builder-debug.tgz | tar -xvf-
- Ex: gdb -k kernel.debug vmcore.0
42. Coredump analysis daemon crash
- 1) uncompress the freaking core *.tgz file
- gunzip < cosd.core-tarball.2.tgz | tar -xvf - cosd.core.0
juniper.conf messages cosd.info.0 juniper.conf.1.gz
- 2) Where is the symbol file by doing what
- bash-2.05b$ what cosd.core.0
- COSD release 7.3R3.6 built by builder on 2006-02-01 08:03:43
UTC
- xathanon.juniper.net
:/build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd
- getsubopt.c 8.1 (Berkeley) 6/4/93
- Copyright (c) 1994 Powerdog Industries.All rights
reserved.
43. Coredump analysis daemon crash
- -bash-2.05b$ gdb
/build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd
cosd.core.0
- Copyright 1998 Free Software Foundation, Inc.
- -bash-2.05b$ gdb
/build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd
cosd.core.0
- Copyright 1998 Free Software Foundation, Inc.
- Core was generated by `cosd'.
- Program terminated with signal 11, Segmentation fault.
- /usr/lib/libisc.so.2: No such file or directory.
- #00x806d6f2 in cos_ifd_configure (dop=0x81e4300,
conf=0x81ba000,
- name=0xbfbff850 "ge-0/3/0", match_len=10, wc_match=0 '00',
- ifd_has_ieee_classifier=1 '01', errmsg=0xbfbffc70 "",
errmsglen=256)
- at
../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705
- 2705cos_ifd->if_flags |=
COS_IFD_CONF_F_IEEE_CLASSIFIER;
- #00x806d6f2 in cos_ifd_configure (dop=0x81e4300,
conf=0x81ba000,
- name=0xbfbff850 "ge-0/3/0", match_len=10, wc_match=0 '00',
- ifd_has_ieee_classifier=1 '01', errmsg=0xbfbffc70 "",
errmsglen=256)
- at
../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705
- #10x806f851 in cos_config_interfaces (dop=0x81e4280,
conf=0x81ba000,
- errmsg=0xbfbffc70 "", errmsglen=256)
- at
../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:3944
#20x807bb53 in cos_config (conf=0x81ba000, errmsg=0xbfbffc70 "",
errmsglen=256) at
../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10816
#30x807be0e in cosd_parse_config (cos_conf=0x81ba000, check_only=0
'00') at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10924
#40x8069ac4 in main (argc=1, argv=0xbfbffe0c) at
../../../../src/juniper/usr.sbin/cosd/cosd_main.c:330 (gdb)l 2700}
else { 2701cos_ifd = cos_pat_to_ifd(pnode); 2702} 2703 2704if
(ifd_has_ieee_classifier) { 2705cos_ifd->if_flags |=
COS_IFD_CONF_F_IEEE_CLASSIFIER; 2706} 2707 2708/* 2709* in commit
check, cosd hasn't built its interface data 44. Coredump analysis
Software or Hardware issues?
- Panic, TLB Data miss, Data accessetc type of system
exceptions:most probably software related. What you should do is to
enable the coredump on the chassisd and gather all the base
information mentioned above.
- pci parity error being reported on the CPU DRAM address space,
this means that this is
- a bogus pci error. The reason is, there is no pci bus connected
to the CPU DRAM.
- Action: In this case, we have to enable the coredump on
chassisd and get the coredump of the PFE component along with the
base information. No RMA should be issued.
- mpc106 machine check caused by error on the PCI Bus
- mpc106 error detect register 1: 0x08, 2: 0x00
- mpc106 error ack count = 2
- mpc106 error address: 0x001d0048 < belongs to CPU DRAM
- mpc106 PCI bus error status register: 0x02
- mpc106 was the PCI master
- C/BE bits: I/O read [0b0010]
- mpc106 error detection reg1: PCI cycle
- mpc106PCI status reg: parity error< parity error.
45. Coredump analysis Software or Hardware issues?
- There is parity protection enabled (ECC is disabled) on the
CPU
- DRAM, if a hw failure occurs here, the message that you
should
- see is: "memory parity/ECC error".
- Action: Run the memory diagnostics tests and RMA.
- mpc106 machine check caused by error on the Processor Bus <
reported by Processor Bus
- mpc106 error detect register 1: 0x04, 2: 0x00
- mpc106 error ack count = 0
- mpc106 error address: 0x02f39e18
- mpc106 Processor bus error status register: 0x72
- transfer type 0b01110, transfer size 2
- mpc106 error detection reg1:memory parity/ECC error< parity
error.
- mpc106 PCI status reg: parity error
46. Coredump analysis Software or Hardware issues?
- There is parity protection enabled (ECC is disabled) on the
CPU
- DRAM, if a hw failure occurs here, the message that you
should
- see is: "memory parity/ECC error".
- Action: Run the memory diagnostics tests and RMA.
- mpc106 machine check caused by error on the Processor Bus <
reported by Processor Bus
- mpc106 error detect register 1: 0x04, 2: 0x00
- mpc106 error ack count = 0
- mpc106 error address: 0x02f39e18
- mpc106 Processor bus error status register: 0x72
- transfer type 0b01110, transfer size 2
- mpc106 error detection reg1:memory parity/ECC error< parity
error.
- mpc106 PCI status reg: parity error
47. Monitoring - logs
- Step 1: configure logging file
- Step 3: monitor start message
- lab@falcons> monitor start mike-isis
- lab@falcons> monitor start messages
- Feb5 20:05:53.517506 Updating LSP falcons.00-00 in
database
- Feb5 20:05:53.517654 Updating L2 LSP falcons.00-00 in TED
48. Booting up system
- request system snapshot partition as-primary
- request system reboot media usb-when reboot from another media,
all the file systems will be under this media.
- request system snapshot part as-primary media
compact-flash
- request system reboot media compact
- request system software add
/var/tmp/junojseries-8.4R2.4-domestic.tgz no-validate
- Request system snapshot--make a image at another storage(if you
are using disk, this will mirror the image to CF. If you are using
CF, this will makes an image at disk.
- request system software delete backup
- request system storage cleanup
- To remove swap space at the compact-flash:
-
http://www.juniper.net/techpubs/software/junos/junos85/rn-sw-85
49. Tools and quick reference
- /volume/build - junos releases and source code. After 8.4, go
to extra hierarchy /volume/build/junos. For example:
/volume/build/junos/8.4/release/8.4R2.4/ship
- http://www- in.juniper.net/eng/cvs_pdf /
- https:// deepthought.juniper.net /app/
- http://cvs/cgi-bin/viewcvs.cgi /
- http:// confluence.jnpr.net /
- /volume/current - cvs functional specs
- http://rogers.jtac-emea.jnpr.net/wiki/index.php?title =
Enginee
50. How to find out what syslog means?
- help syslog SNMPD_SUBAGENT_NO_RESOURCES Name:
SNMPD_SUBAGENT_NO_RESOURCES Message: No resources available for
subagent (): Help: Subagent resources were temporarily exhausted
Description: The SNMP agent process (snmpd) uses certain resources
for communication with subagents. Resources were not available for
communication with the indicated subagent. Type: Error: An error
occurred Severity: notice Cause: An internal software failure
occurred. Action: Contact your technical support
representative.
51. How to find out the data between 2 proc sockets?
- Find out the processes ID (use snmpd and mib2d as example)
- root@Kelly_RE0% ps -aux | egrep -i "snmpd|mib2d"
- root83220.00.25036 3932??S4Feb080:12.24 /usr/sbin/snmpd -N
- root83020.00.24464 3892??I4Feb080:10.35 /usr/sbin/mib2d N
- root@Kelly_RE0% fstat -p 8302
- USERCMDPIDFD MOUNTINUM MODESZ|DV R/W
- rootmib2d830217* local stream faab6c80 fab03e60
- root@Kelly_RE0% fstat -p 8322
- USERCMDPIDFD MOUNTINUM MODESZ|DV R/W
- rootsnmpd832215* local stream fab03e60 faab6c80
- 3. Then, check the socket data.
- root@Kelly_RE0% netstat -Aan | egrep -i "mib2d|snmpd|Send"
- PCBProto Recv-Q Send-QLocal AddressForeign Address(state)
- PCBProto Recv-Q Send-QLocal AddressForeign Address(state)
- AddressTypeRecv-Q Send-QInodeConnRefsNextref Addr
- f5f4e6c0 stream000 faad35a000 /var/run/snmpd_stream
- f5f4b300 stream000 faa47aa000 /var/run/snmpd_stream
- f5f4fc20 stream000 fab67dc000
52. How to do RMA?
53. Trouble shoot T-series
- show pfe statistics traffic
- show interface [int] extensive
- show lchip [x] lout sw lsif
- show lchip [x] lout sw desrd
- show lchip [x] lout sw hdrf
- show lchip [x] lout sw nlif
- show lchip [x] lout hw lsif
- show lchip [x] lout hw nlif
- show lchip [x] lout hw hdrf
show lchip [x] lout hw nlifshow lchip [x] stream [stream_#]show
lchip [x] lout registers lsif lsif [stream_#]( where [stream_#] is
the stream you found which corresponds to the interface that has
the problem using the show lchip ifd output above )show lchip [x]
lout registers nlif nlif 54. Trouble shoot T-series
- show lchip [x] lout sw lsif
- show lchip [x] lout sw desrd
- show lchip [x] lout sw hdrf
- show lchip [x] lout sw nlif
- show lchip [x] lout hw lsif
- show lchip [x] lout hw nlif
- show lchip [x] lout hw hdrf
- show lchip [x] lout hw nlif
- show lchip [x] stream [stream_#]
- show lchip [x] lout registers lsif lsif [stream_#]
- (where [stream_#] is the stream you have seen on the "show
lchip ifd"
- output under the lchip [x])
- show lchip [x] lout registers nlif nlif
- show lchip [x] lout reg nlif dbufpart
- show lchip [x] lout reg nlif bdispmon
- Wait a little, hopefully after a few more errors go by.
- show mq [x] wan stream active stat
- Show chassis fabric topology Show chassis fabric sibs Show
chassis fabric fpcs
55. How to trouble shoot SNMP and MIB2d
- show snmp statistics extensive
- show system virtual-memory
56. How to trouble shoot routing and forwarding issues?
- FPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix
192.12.1.2
- IPv4 Route Table 0, default.0, 0x0:
- DestinationNH IP AddrTypeNH ID Interface
- --------------------------------- --------------- --------
----- ---------
- 192.12.1.2Hold716 ge-7/0/4.0
57. How to trouble shoot routing and forwarding issues?
- install@FED1DSRJ01-LAB-re0> show route forwarding-table
destination 192.12.1.2
- DestinationType RtRef Next hopType Index NhRef Netif
- 192.12.1.2/32dest1 192.12.1.2hold7162 ge-7/0/4.0
- Routing table: __juniper_private1__.inet
- DestinationType RtRef Next hopType Index NhRef Netif
- Routing table: __juniper_private2__.inet
- DestinationType RtRef Next hopType Index NhRef Netif
- Routing table: FED1J1MIS.inet
- DestinationType RtRef Next hopType Index NhRef Netif
- Routing table: TEST-L3VPN.inet
- DestinationType RtRef Next hopType Index NhRef Netif
58. How to trouble shoot routing and forwarding issues?
- install@FED1DSRJ01-LAB-re0> show arp
- MAC AddressAddressNameInterfaceFlags
- 02:01:00:00:00:05 10.0.0.510.0.0.5em0.0none
- 00:04:80:9d:b5:00 10.1.1.110.1.1.1fxp0.0none
- 00:0c:29:9a:e5:38 10.1.1.11510.1.1.115fxp0.0none
- 00:05:85:9b:5d:f5 31.1.1.231.1.1.2ge-7/0/3.493 none
- 00:14:f6:56:b8:7e 68.1.0.20468.1.0.204ge-7/1/0.0none
- 02:01:00:00:00:05 128.0.0.5128.0.0.5em0.0none
- 00:00:c0:10:01:02 192.16.1.2192.16.1.2ge-7/0/5.0none
59. How to trouble shoot routing and forwarding issues?
- install@FED1DSRJ01-LAB-re0> show arp
- MAC AddressAddressNameInterfaceFlags
- 02:01:00:00:00:05 10.0.0.510.0.0.5em0.0none
- 00:04:80:9d:b5:00 10.1.1.110.1.1.1fxp0.0none
- 00:0c:29:9a:e5:38 10.1.1.11510.1.1.115fxp0.0none
- 00:05:85:9b:5d:f5 31.1.1.231.1.1.2ge-7/0/3.493 none
- 00:14:f6:56:b8:7e 68.1.0.20468.1.0.204ge-7/1/0.0none
- 02:01:00:00:00:05 128.0.0.5128.0.0.5em0.0none
- 00:00:c0:10:01:02 192.16.1.2192.16.1.2ge-7/0/5.0none
60. How to trouble shoot routing and forwarding issues?
- install@FED1DSRJ01-LAB-re0> show route protocol ospf
- inet.0: 260 destinations, 387 routes (186 active, 0 holddown,
77 hidden)
- @ = Routing Use Only, # = Forwarding Use Only
- + = Active Route, - = Last Active, * = Both
- 0.0.0.0/0*[OSPF/10] 09:25:03, metric 16777215
- 3.1.1.0/24*[OSPF/150] 09:23:28, metric 0, tag 0
- 10.1.0.0/16[OSPF/150] 09:23:28, metric 0, tag 0
- 10.1.1.0/24[OSPF/150] 09:23:28, metric 0, tag 0
- 10.1.200.0/28[OSPF/150] 09:23:28, metric 0, tag 0
- 10.99.0.0/16[OSPF/150] 09:23:28, metric 0, tag 0
- 10.99.99.0/24[OSPF/150] 09:23:28, metric 0, tag 0
- 24.234.6.0/24*[OSPF/10] 00:54:30, metric 182
- > to 68.1.0.204 via ge-7/1/0.0
- 24.234.6.0/27*[OSPF/10] 00:54:30, metric 166
- > to 68.1.0.204 via ge-7/1/0.0
- 24.248.129.0/27[OSPF/150] 09:23:28, metric 0, tag 0
61. How to trouble shoot routing and forwarding issues?
- FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix
192.12.1.2
- IPv4 Route Table 0, default.0, 0x0:
- DestinationNH IP AddrTypeNH ID Interface
- --------------------------------- --------------- --------
----- ---------
- 192.12.1.2192.12.1.2Unicast716 ge-7/0/4.0
- FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip lookup
192.12.1.2
- Route Information (192.12.1.2):
- interface : ge-7/0/4.0 (87)
- Nexthop prefix : 192.12.1.2
- FFPC7(FED1DSRJ01-LAB-re0 vty)#
62. How to trouble shoot routing and forwarding issues?
- install@FED1DSRJ01-LAB-re0> show interfaces filters
ge-7/0/4
- InterfaceAdmin Link Proto Input FilterOutput Filter
- FFPC7(FED1DSRJ01-LAB-re0 vty)# show nhdb interface
ge-7/0/4
- IDTypeInterfaceNext Hop AddrProtocolEncapMTU
-
-------------------------------------------------------------------
- 625Bcastge-7/0/4.0-IPv4Ethernet0
- 626Receivege-7/0/4.0192.12.1.0IPv4Ethernet0
- 628Resolvege-7/0/4.0-IPv4Ethernet0
- 716Unicastge-7/0/4.0192.12.1.2IPv4Ethernet1514
63. Lab stuff
- Agilent Router Tester. Remote access:
- Top 3 chassis: 172.19.59.28
- Bottom 3 chassis: 172.19.58.12
- For FE, need to config SFP
- IXIA: VNC 172.19.58.2 (SV) 172.25.84.219(HD)
- IXIA application server: 172.19.58.17
64. How to trouble shoot EOAM?
-
http://www.juniper.net/techpubs/software/junos/junos82/swconfig82-network-interfaces/html/interfaces-ethernet-config50.html#1272612
-
http://www.juniper.net/techpubs/software/junos/junos82/swconfig82-network-interfaces/html/interfaces-summary298.html#11618684
65. How to trouble shoot EOAM?
66. How to trouble shoot EOAM?
67. How to Manually mount a USB/CF storage?
- http://kb.juniper.net/KB8017
- First upload the desired JUNOS image to the router via ftp to
/var/tmp.
- Connect the USB mass storage device.
- Format the USB device by dropping to shell (start shell) then
enter "dd if=/dev/zero of=/dev/da0 bs=128k" (root access required).
Note this step can take several minutes to complete with no output
to the CLI window.
- Label the device by entering "disklabel -r -w da0 auto".(!! if
you move the USB/CF around, you need to execut this command before
mounting)
- Create the file system with "newfs -U /dev/da0c".
- Create a dir to be used as a mount point with "mkdir
/var/tmp/usb".
- Mount the USB device using "mount /dev/da0c /var/tmp/usb".
- df -h can be used to verify the mount.
- Copy the JUNOS install image to the USB device.
- cp /var/tmp/junos-jseries-8.0R2.8-domestic.tgz
/var/tmp/usb
- Delete the original image to free up space on the CF.
- rm /var/tmp/junos-jseries-8.0R2.8-domestic.tgz
- Use the "request system software add
/var/tmp/usb/junos-jseries-8.0R2.8-domestic.tgz" command to install
the new JUNOS version.
68. How to do tcpdump at Junos?
- You have to login as root
- You have to know which incoming interface?
- root@bananas-re0% tcpdump -xvf -i so-1/1/0
69. Ethernet OAM
- Ethernet OAM types In short, there are two types of Ethernet
OAM: 1. Ethernet OAM as defined by 802.3ah This is referred as LFM
(Link Fault Management) and are identified by the ether-type 0x8809
(slow protocol type packets), sub-type 3. 2. Ethernet OAM as
defined by IEEE 802.1ag This is referred as CFM (Connectivity Fault
Management) and can be by the ether-type 0x8902.
- Ethernet OAM implementation in JunOS Ethernet OAM is
implemented using the RE user space daemons "lfmd" and "cfmd".
Also, both "lfmd" and "cfmd" use the "ppmd" daemon on the PFE for
some periodic packet processing. There is a packet processing path
in the RE kernel as well in addition to the daemons mentioned
above.
70. Ethernet OAM
- Ethernet OAM for regular Ethernet interfaces Both 802.3ah (LFM)
and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the
regular Ethernet interfaces with the following restrictions.
802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs
and NOT on the Ethernet IFLs. Also, these packets are always VLAN
untagged. However, 802.1ag (CFM) type OAM can be configured either
on an Ethernet IFD or IFL. If this is configured on an IFD, the
packets will be always VLAN untagged. If this is configured on an
IFL, it will be either VLAN tagged or untagged based on the
"vlan-tagging" keyword configuration on an Ethernet IFD.
71. Ethernet OAM
- Link monitoring in Ethernet OAM detects and indicates link
faults under a variety of conditions. Link monitoring uses the
event notification OAMPDU and sends events to the remote OAMentity
when there are problems detected on the link. The error events
include the following:
- Error Symbol Period (error symbols per second)The number of
symbol errors that occurred during a specified period exceeded a
threshold. These errors are coding symbol errors.
- Error Frame (error frames per second)The number of frame errors
detected during a specified period exceeded a threshold.
- Error Frame Period (error frames pernframes)The number of frame
errors within the last n frames has exceeded a threshold.
- Error Frame Seconds Summary (error seconds permseconds)The
number of error seconds (1-second intervals with at least one frame
error) within the last m seconds has exceeded a threshold. Since
IEEE 802.3ah OAM does not provide a guaranteed delivery of any OAM
PDU, the event
- notification OAM PDU may be sent multiple times to reduce the
probability of a lost notification. A sequence number is used to
recognize duplicate events
72. Ethernet OAM
- Ethernet OAM for regular Ethernet interfaces Both 802.3ah (LFM)
and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the
regular Ethernet interfaces with the following restrictions.
802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs
and NOT on the Ethernet IFLs. Also, these packets are always VLAN
untagged. However, 802.1ag (CFM) type OAM can be configured either
on an Ethernet IFD or IFL. If this is configured on an IFD, the
packets will be always VLAN untagged. If this is configured on an
IFL, it will be either VLAN tagged or untagged based on the
"vlan-tagging" keyword configuration on an Ethernet IFD.
73. Ethernet OAM one scenario ( 2008-0401-0623)
- Scenario: Two T640s with JUNOS 8.2SR are connected together
through an optical transport network (e.g., Fujitsu 7500/7600),
using LAN-PHY on 10GE IQ2 PICs.
- Question: If there is a link failure in the transport network
and the 10GE links between the Fujitsu switches and the T640s stay
up, will the Local T640 send out Ethernet 802.3ah OAMPDUs with the
Flags for Critical Link Events(1) and the Link Event TLVs(2) to the
Remote T640?
- Answer: No. None of that will happen. What will happen is, the
OAM Discovery INFO PDUs will timeout and both sides will detect
that and mark a failure on their respective links. If only one
direction of the link is down, one side will be in "Active Send
Local" state and the other side will be in "Send Local Remote"
state. There is no reason to send Link Event TLVs in the above
situation as it's a link fault, not a framing error.
- The reason we do not send Link-Fault or Dying Gasp is, by the
time we detect a Rx fault, the ifd is marked down and the Tx is
also brought down. The Critical Event is not defined in the 802.3ah
for any specific purposes,and is implementation dependant. In
Juniper implementation, we use Critical event to simulate RDI
functionality. We only send Critical event in case we have a
CCC-DOWN on the ifls on the interface marked by RPD and an action
profile to send a critical event is defined.
74. Ethernet OAM one scenario ( 2008-0401-0623)
- ( world-readable| no- world-readable );} console{ facility
severity ;} file filename{ facility severity ; explicit-priority ;
match" regular-expression "; archive{ files number ; size size ;(
world-readable| no- world-readable );}} host( hostname|
other-routing-engine | scc-master) { facility severity ;
explicit-priority ; facility-override facility ; log-prefix string
; match" regular-expression ";} source-address source-address ;
time -format(year | millisecond | year millisecond); user(
username| *) { facility severity ; match" regular-expression
";}}
75. CoS configuration ( 2008-0523-0448)
-
http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/frameset.html
- In the following classifier example, packets with EXP bits 000
are assigned to the data-queue forwarding class with a low loss
priority, and packets with EXP bits 001 are assigned to the
data-queue forwarding class with a high loss priority.
-
-
- forwarding-class data-queue {
-
-
-
- loss-priority low code-points 000;
-
-
-
- loss-priority high code-points 001;
- In the following drop-profile map example, the scheduler
includes two drop-profile maps, which specify that packets are
evaluated by the low-drop drop profile if they have a low loss
priority and are from any protocol. Packets are evaluated by the
high-drop drop profile if they have a high loss priority and are
from any protocol.
-
-
- drop-profile-map loss-priority low protocol any drop-profile
low-drop;
-
-
- drop-profile-map loss-priority high protocol any drop-profile
high-drop;
- In the following rewrite rule example, packets in the be
forwarding class with low loss priority are assigned the EXP bits
000, and packets in the be forwarding class with high loss priority
are assigned the EXP bits 001.
-
-
-
- loss-priority low code-point 000;
-
-
-
- loss-priority high code-point 001;
76. How to verify packages are corrupted?
- root@% cd /altroot/packages/
- SHA1 (jbase-8.5R3.4) =
51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c
- SHA1 (jdocs-8.5R3.4) =
c56296f2016d5ddbf8b22c00cb8c06dc5c664271
- SHA1 (jkernel-8.5R3.4) =
fedc82d6e8edb6b5ff972ac4c0f22885841ee48e
- SHA1 (jpfe-T-8.5R3.4) =
f8ea2b28cf27a168a1023b0e544cdfb047ac2f0e---> corrupted
- SHA1 (jpfe-common-8.5R3.4) =
0034ccbd5bd1b2bbd9b9ee41d3b42c50443e5562---> corrupted
- SHA1 (jroute-8.5R3.4) =
5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f
- root@% sha1 /packages/j*8.5R3.4
- SHA1 (/packages/jbase-8.5R3.4) =
51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c
- SHA1 (/packages/jdocs-8.5R3.4) =
c56296f2016d5ddbf8b22c00cb8c06dc5c664271
- SHA1 (/packages/jkernel-8.5R3.4) =
fedc82d6e8edb6b5ff972ac4c0f22885841ee48e
- SHA1 (/packages/jpfe-T-8.5R3.4) =
f14de1eb8e537a35088864192d6838bb24804492
- SHA1 (/packages/jpfe-common-8.5R3.4) =
270c4f4cc9c0afb6ba52c6916c2213eeba851ddc
- SHA1 (/packages/jroute-8.5R3.4) =
5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f
77. Class-of-Service trouble shooting
- There is bug in Gimlet FPC where the PLP high defined at
classifier will *NOT* be copied to notification. Thus if egress FPC
might have rewrite rule messed up.
- Gimlet FPC to Gimlet FPC has no problem.
- Gimble FPC to Stoli FPC has problem
- Gimlet FPC to Gimlet FPC with drop-profile has problem.
- To work around this problem for scenario 2 & 3:
- lab@slayer-re1# set class-of-service copy-plp
- Default forwarding class:
78. Class-of-Service trouble shooting
-
http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/swconfig-cos.pdf
- Table 43: Default MPLS EXP Rewrite Table(P230)
- ------------------------------------------------
- Forwarding ClassLoss PriorityCoS Value
- expedited-forwarding(1)low010
- expedited-forwardinghigh011
- assured-forwarding(2)low100
- assured-forwardinghigh101
79. Class-of-Service trouble shooting
-
http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/swconfig-cos.pdf
- Table 42: Default Packet Header Rewrite Mappings (p225)
- Map from Forwarding ClassPLP ValueMap to DSCP/DSCP IPv6/
EXP/IEEE/IP
- expedited-forwardinglowef
- expedited-forwardinghighef
- assured-forwardinglowaf11
- assured-forwardinghighaf12 (DSCP/DSCP IPv6/EXP)
- network-controllownc1/cs6
- network-controlhighnc2/cs7
- The mapping of alias to EXP code point is at next slide. Same
thing to look up alias to DSCP code point.
80. Class-of-Service trouble shooting
- lab@slayer-re1> show class-of-service code-point-aliases
exp
81. PLP Treatment on LMNR Platforms Overview 82. Problem
- Customer Cox was seeing an increase of Non-Real-Time class
traffic in the network when replacing IQ2 10GE PICs by 10GE XENPAK
(non-IQ2) PICs.
- Hard to isolate as there was a mix of traffic from different
sources.
- Initially though the problem was due to missclasification.
83. Topology IP unlabeled Traffic IP unlabeled Traffic LSP
xe-0/1/0 84. Configuration: Forwarding Classes
- > ...service forwarding-classes
85. Configuration: IP-Prec. Classifier
- forwarding-class BEST-EFFORT {
- loss-priority high code-points BEST-EFFORT-be;
- forwarding-class NON-REAL-TIME {
- loss-priority high code-points NON-REAL-TIME-af11;
- forwarding-class INTERACTIVE {
- loss-priority low code-points INTERACTIVE-af21;
- forwarding-class REAL-TIME {
- loss-priority low code-points REAL-TIME-af31;
- loss-priority low code-points VIDEO-af41;
- loss-priority low code-points VOICE-ef;
- forwarding-class NETWORK-CONTROL {
- loss-priority low code-points NETWORK-CONTROL-nc1;
inet-precedence { BEST-EFFORT-be 000; NON-REAL-TIME-af11 001;
INTERACTIVE-af21 010; REAL-TIME-af31 011; VIDEO-af41 100; VOICE-ef
101; NETWORK-CONTROL-nc1 110; } 86. Configuration: EXP
Classifier
- forwarding-class BEST-EFFORT {
- loss-priority high code-points BEST-EFFORT-be;
- forwarding-class NON-REAL-TIME {
- loss-priority high code-points NON-REAL-TIME-af11;
- forwarding-class INTERACTIVE {
- loss-priority low code-points INTERACTIVE-af21;
- forwarding-class REAL-TIME {
- loss-priority low code-points REAL-TIME-af31;
- loss-priority low code-points VIDEO-af41;
- loss-priority low code-points VOICE-ef;
- forwarding-class NETWORK-CONTROL {
- loss-priority low code-points NETWORK-CONTROL-nc1;
BEST-EFFORT-be 000; NON-REAL-TIME-af11 001; INTERACTIVE-af21
010; REAL-TIME-af31 011; VIDEO-af41 100; VOICE-ef 101;
NETWORK-CONTROL-nc1 110; 87. Configuration: Rewrite Rules, EXP
- forwarding-class BEST-EFFORT {
- loss-priority low code-point BEST-EFFORT-be;
- loss-priority high code-point BEST-EFFORT-be;
- forwarding-class NON-REAL-TIME {
- loss-priority low code-point NON-REAL-TIME-af11;
- loss-priority high code-point NON-REAL-TIME-af11;
- forwarding-class INTERACTIVE {
- loss-priority low code-point INTERACTIVE-af21;
- loss-priority high code-point INTERACTIVE-af21;
- forwarding-class REAL-TIME {
- loss-priority low code-point REAL-TIME-af31;
- loss-priority high code-point REAL-TIME-af31;
- loss-priority low code-point VIDEO-af41;
- loss-priority high code-point VIDEO-af41;
- loss-priority low code-point VOICE-ef;
- loss-priority high code-point VOICE-ef;
- forwarding-class NETWORK-CONTROL {
- loss-priority low code-point NETWORK-CONTROL-nc1;
- loss-priority high code-point NETWORK-CONTROL-nc1;
88. PLP handling 89. Which PLP ?
- The L to N notification cell contains two bits (three with
tri-color marking) of interest:
- The pseudo-plp bit: This is bit 2 of the QoS field (6-bits),
and its used by the Lin BA Classifier and Rewrite rules
- The real plp bit: this is a separate bit, see the Lin
functional description for location.
90. PLP On LMNR 91. Example: IP packet, precedence 000, non-IQ2
PIC
- Lets say we receive a packet with IP-Prec bits 000.Lets say we
have a BA Classifier that classifies IP-Prec: 000 as Best-Effort
(queue 0) and plp=high:
- # show class-of-service code-point-aliases inet-precedence
92. Contd
- # show class-of-service classifiers inet-precedence
CLASSIFY-IPP
- forwarding-class BEST-EFFORT {
- loss-priority high code-points 000;
- # show class-of-service forwarding-classes
93. Ctd
- Because this packets real-plp bit will remain 0, RED will treat
it as such.If we have the following rewrite rule:
- apena@austinp-re0# show class-of-service rewrite-rules
- forwarding-class BEST-EFFORT {
- loss-priority low code-point 000;
- loss-priority high code-point 000; 111.0.0.1: ICMP echo
request, id 51991, seq 6, length 64 19:03:10.506285 Out SERVICES
service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 >
101.1.1.1: ICMP echo reply, id 51991, seq 6, length 64
19:03:11.507050In IP 101.1.1.1 > 111.0.0.1: ICMP echo request,
id 51991, seq 7, length 64 19:03:11.507061 Out SERVICES service id
64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1:
ICMP echo reply, id 51991, seq 7, length 64 19:03:12.507977In IP
101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 8,
length 64 19:03:12.507988 Out SERVICES service id 64 flags 0x82
service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo
reply, id 51991, seq 8, length 64 19:03:13.508794In IP 101.1.1.1
> 111.0.0.1: ICMP echo request, id 51991, seq 9, length 64
19:03:13.508802 Out SERVICES service id 64 flags 0x82 service set
id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991,
seq 9, length 64 19:03:14.509561In IP 101.1.1.1 > 111.0.0.1:
ICMP echo request, id 51991, seq 10, length 64 101. IPSec
configuration and troubleshooting
- lab@jazz-re0# run show log kmd
- Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared
- Jul 17 18:33:26 Initialising the KMD ipsec-interface-id
pool
- Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to
kernel
- Jul 17 18:33:26 Initializing certificate manager
- Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 18:57:25 Initialising the KMD ipsec-interface-id
pool
- Jul 17 18:57:38 Initialising the KMD ipsec-interface-id
pool
- Jul 17 18:58:53 Initialising the KMD ipsec-interface-id
pool
- Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to
kernel
- Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to
kernel
102. IPSec configuration and troubleshooting
- lab@jazz-re0# run show log kmd
- Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared
- Jul 17 18:33:26 Initialising the KMD ipsec-interface-id
pool
- Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to
kernel
- Jul 17 18:33:26 Initializing certificate manager
- Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 18:57:25 Initialising the KMD ipsec-interface-id
pool
- Jul 17 18:57:38 Initialising the KMD ipsec-interface-id
pool
- Jul 17 18:58:53 Initialising the KMD ipsec-interface-id
pool
- Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to
kernel
- Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC
index=0 Interface name: sp-0/0/0 Length:1392 to kernel
- Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to
kernel
103. How to compare rollback?
- rprivette@CHRL-HAGG-03> show system rollback compare 0
2
- [edit interfaces ge-3/3/1 unit 3478]
- -description "16/VLXX/010009/TWCS - FREEMAN WHITE # 255277
[ENLAN]";
- +description "16/KDFN/010010/TWCS - Freeman White #
FW115671";
- -encapsulation vlan-vpls;
104. MX VLAN configuration what are the new stuff?
- MISTP: Cisco Proprietary Multiple Instance STP
- PVST+:Per-VLAN spanning-tree plus
105. MX VLAN Trunking configuration General guideline
- Generally, there are four things that you must configure in an
L2 environment:
- Interfaces and virtual LAN (VLAN) tagsL2 interfaces are usually
various type of Ethernet links with VLAN tags used to connect to
customer devices or other bridges or routers.
- Bridge domains and virtual switchesBridge domains limit the
scope of media access control (MAC) learning (and thereby the size
of the MAC table) and also determine where the device should
propagate frames sent to broadcast, unknown unicast, and multicast
(BUM) MAC addresses. Virtual switches allow for the configuration
of multiple, independent bridge domains.
- Spanning Tree Protocols (xSTP, where the x represents the STP
type)Bridges function by associating a MAC address with an
interface, similar to the way a router associates an IP network
address with a next-hop interface. Just as routing protocols use
packets to detect and prevent routing loops, bridges use xSTP
frames to detect and prevent bridging loops. (L2 loops are more
devastating to a network because of the broadcast nature of
Ethernet LANs.)
- Integrated bridging and routing (IRB)Support for both Layer 2
bridging and Layer 3 routing on the same interface. Frames are
bridged if they are not sent to the router's MAC address. Frames
sent to the router's MAC address are routed to other interfaces
configured for Layer 3 routing.
106. MX VLAN Trunking configuration vlan tagging
-
- encapsulationflexible-ethernet-services;
-
- vlan-tagging; # Customer interface uses singly-tagged
frames
-
-
- encapsulation vlan-bridge;
-
- encapsulationextended-vlan-bridge;
107. MX VLAN Trunking configuration bridge domain
- Configure the virtual switches and bridge domains on all three
routers. There is always a default virtual switch in the router for
L2 functions; however, if there is only one L2 network, then the
virtual switch instance type is not needed.
- Configure a bridge domain on Router 1:
108. MX VLAN Trunking configuration MSTP-1
- MSTI:Multiple Spanning Tree Instances
- CIST:Common and Internal Spanning Tree
- MSTP:Multiple Spanning Tree Protocol
- Configuration name:The names must match to be in the same
region
- Revision Level:must be the same across the same region.
- VLAN-to-MSTI mapping:vlans mapped to this MSTP instance.
109. MX VLAN Truncking configuration MSTP-2
-
-
- configuration-name mstp-for-R1-2-3; # The names must match to
be in the same region
-
-
- revision-level 3; # The revision levels must match
-
-
- bridge-priority 0; # This bridge acts as root bridge for VLAN
100 and 200
-
-
-
- vlan100; # This VLAN corresponds to MSTP instance 1
-
-
-
- vlan200; # This VLAN corresponds to MSTP instance 2
110. MX VLAN Truncking configuration IRB-1
- You configure IRB in two steps:
- (1) Configure the IRB interface using the irb statement.
- (2) Reference the IRB interface at the bridge domain level of
the configuration.
- IRB supports Layer 2 bridging and Layer 3 routing on the same
interface. If the MAC address on the arriving frame is the same as
that of the IRB interface, then the packet inside the frame is
routed. Otherwise, the MAC address is learned or looked up in the
MAC address database.
111. MX VLAN configuration IRB-2
-
-
-
- address 10.0.10.2/24; # Routing interface
-
-
-
-
- virtual-address 10.0.1.51;
-
-
-
-
- virtual-address 10.0.2.51;