Sooooo long since last time, why you my ask? Well, ehm, one word: baby!
Back to topic, I have this 2110 network running for a while, I have a blog post coming soon on that (part 2). I have some issues, a lot of CRC, Error, Symbol etc. This are coming and going. Such pain to troubleshoot.
So lets deep dive more on this. I have setup Telemetry for this and Prometheus are scraping this, the data are presented in Grafana for nice logs and alerting. That is fine and good, but now I started to get some alerts telling me that there are errors on some of the links. Grafana did not lie, the counters under the interface said the same thing.
Spine2#show int eth61/1
Ethernet61/1 is up, line protocol is up (connected)
Hardware is Ethernet, address is 985d.821f.b84d
Description: Leaf2 - Ethernet29/1
IPv6 link-local address is fe80::9a5d:82ff:fe1f:b84d/64
Address being determined by SLAAC
No IPv6 global unicast address is assigned
IP MTU 9194 bytes , BW 100000000 kbit
Full-duplex, 100Gb/s, auto negotiation: off, uni-link: n/a
Up 5 days, 19 hours, 42 minutes, 36 seconds
Loopback Mode : None
0 link status changes since last clear
Last clearing of "show interface" counters 5 days, 19:40:27 ago
30 seconds input rate 33.3 kbps (0.0% with framing overhead), 54 packets/sec
30 seconds output rate 60.9 kbps (0.0% with framing overhead), 87 packets/sec
173735316708 packets input, 217253077423021 bytes
Received 3 broadcasts, 173710958446 multicast
4929 runts, 0 giants
22052 input errors, 17123 CRC, 0 alignment, 18120 symbol, 0 input discards
0 PAUSE input
72666851232 packets output, 91623591182571 bytes
Sent 1 broadcasts, 72642461862 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 0 output discards
0 PAUSE outputThe next thing here was to look at the dbm values for the QSFP, they looked good.
Spine2#show int eth61/1 transceiver
If device is externally calibrated, only calibrated values are printed.
N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).
Bias Optical Optical
Temp Voltage Current Tx Power Rx Power
Port (Celsius) (Volts) (mA) (dBm) (dBm) Last Update
----- --------- -------- -------- -------- -------- -------------------
Et61/1 41.68 3.27 35.53 1.81 0.23 0:00:03 agoThe next thing was to look at the other end,
Leaf2#show int eth29/1
Ethernet29/1 is up, line protocol is up (connected)
Hardware is Ethernet, address is fcbd.6786.4b2b
Description: Spine2 - Ethernet61/1
IPv6 link-local address is fe80::febd:67ff:fe86:4b2b/64
Address being determined by SLAAC
No IPv6 global unicast address is assigned
IP MTU 9194 bytes , BW 100000000 kbit
Full-duplex, 100Gb/s, auto negotiation: off, uni-link: n/a
Up 6 days, 5 hours, 49 minutes, 59 seconds
Loopback Mode : None
8 link status changes since last clear
Last clearing of "show interface" counters 21 days, 8:07:58 ago
5 minutes input rate 14.3 Gbps (14.6% with framing overhead), 1420090 packets/sec
5 minutes output rate 35.7 Gbps (36.3% with framing overhead), 3568027 packets/sec
1673725141717 packets input, 2111527576144721 bytes
Received 7 broadcasts, 1673613199109 multicast
1 runts, 0 giants
1 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
0 PAUSE input
3696743864070 packets output, 4621816202569007 bytes
Sent 8 broadcasts, 3696632144932 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 0 output discards
0 PAUSE outputLeaf2#show int eth29/1 transceiver
If device is externally calibrated, only calibrated values are printed.
N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).
Bias Optical Optical
Temp Voltage Current Tx Power Rx Power
Port (Celsius) (Volts) (mA) (dBm) (dBm) Last Update
----- --------- -------- -------- -------- -------- -------------------
Et29/1 42.61 3.29 39.67 1.27 0.75 0:00:05 agoEvery thing looked create at this end.
I did do a clean of all fiber and QSFP, but stil the same. I change the QSFP at both ends and the same, did also recode the QSFP with Arista. No luck!
What to do when stuck? Ask my “friend” google. I came over a command:
Spine2#show int eth61/1 phy detail
Current System Time: Wed Sep 2 18:39:32 2020
Ethernet61/1
Current State Changes Last Change
------------- ------- -----------
PHY state linkUp 11 6 days, 5:54:15 ago
Interface state up 10 6 days, 5:54:14 ago
HW resets 0 never
Transceiver 100GBASE-CWDM4 6 8 days, 8:28:43 ago
Transceiver SN XXXXX
Oper speed 100Gbps
Interrupt Count 12
Transceiver Reset Count 0 never
Transceiver Interrupt Count 2 8 days, 8:28:40 ago
Transceiver Smbus Failures 0 never
Diags mode normalOperation
Model Bcm56971-tscf16 (A)
Loopback none
Xcvr EEPROM read timeout 0 never
Spurious xcvr detection 0 never
DOM control/status fail 0
Presence indication xcvrPresent 8 8 days, 8:28:50 ago
Bad EEPROM checksums 0 never
RX_LOS since system boot False 0 never
RX_LOS since insertion 0
TX_FAULT since system boot False 0 never
TX_FAULT since insertion 0
PMA/PMD RX signal detect ok 1 128 days, 2:20:43 ago
Forward Error Correction Reed-Solomon
Reed-Solomon codeword size 528
FEC alignment lock ok 1 6 days, 5:54:18 ago
FEC corrected codewords 7565 107992 0:00:03 ago
FEC uncorrected codewords 0 0 never
FEC lane corrected symbols
Lane 0 0 0 never
Lane 1 0 0 never
Lane 2 7571 107992 0:00:03 ago
Lane 3 0 0 never
FEC lane mapping
FEC lane 00 01 02 03
PMA lane 00 01 02 03
PCS RX link status up 9 6 days, 5:54:17 ago
PCS high BER ok 0 never
PCS BER 12 42019 8 days, 8:29:57 ago
PCS error blocks 0 6 days, 5:54:23 ago
Lane 0
RXPPM CLK90 CLKP1 PF(M,L) VGA DCO DFE(1,2,3,4,5,6) TXPPM TXEQ(n1,m,p1,2,3)
0 32 1 0,0 12 2 13, -8, 1, -1, 0, 0 0 7, 68,25, 0, 0Right here we can see that FEC have to correct on Lane 2 and the PCS BER is rising, this means that there are problems on layer 1. .
Overview
Spine2#show int eth61/1 phy detail
Current System Time: Wed Sep 2 18:39:32 2020
Ethernet61/1
Current State Changes Last Change
————- ——- ———–
PHY state linkUp 11 6 days, 5:54:15 ago
Interface state up 10 6 days, 5:54:14 ago
HW resets 0 never
Transceiver 100GBASE-CWDM4 6 8 days, 8:28:43 ago
Transceiver SN XXXXXX
Oper speed 100Gbps
Interrupt Count 12
Transceiver Reset Count 0 never
Transceiver Interrupt Count 2 8 days, 8:28:40 ago
Transceiver Smbus Failures 0 never
Diags mode normalOperation
Model Bcm56971-tscf16 (A)
Loopback none
Xcvr EEPROM read timeout 0 never
Spurious xcvr detection 0 never
DOM control/status fail 0
Presence indication xcvrPresent 8 8 days, 8:28:50 ago
Bad EEPROM checksums 0 never
RX_LOS since system boot False 0 never
RX_LOS since insertion 0
TX_FAULT since system boot False 0 never
TX_FAULT since insertion 0
PMA/PMD RX signal detect ok 1 128 days, 2:20:43 ago
Forward Error Correction Reed-Solomon
Reed-Solomon codeword size 528
FEC alignment lock ok 1 6 days, 5:54:18 ago
FEC corrected codewords 7565 107992 0:00:03 ago
FEC uncorrected codewords 0 0 never
FEC lane corrected symbols
Lane 0 0 0 never
Lane 1 0 0 never
Lane 2 7571 107992 0:00:03 ago
Lane 3 0 0 never
FEC lane mapping
FEC lane 00 01 02 03
PMA lane 00 01 02 03
PCS RX link status up 9 6 days, 5:54:17 ago
PCS high BER ok 0 never
PCS BER 12 42019 8 days, 8:29:57 ago
PCS error blocks 0 6 days, 5:54:23 ago
Lane 0
RXPPM CLK90 CLKP1 PF(M,L) VGA DCO DFE(1,2,3,4,5,6) TXPPM TXEQ(n1,m,p1,2,3)
0 32 1 0,0 12 2 13, -8, 1, -1, 0, 0 0 7, 68,25, 0, 0
Reed-Solomon
I will not rewrite wikipedia here so I just link to the article. https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
Reed–Solomon codes operate on a block of data treated as a set of finite-field elements called symbols. Reed–Solomon codes are able to detect and correct multiple symbol errors. By adding t = n − k check symbols to the data, a Reed–Solomon code can detect (but not correct) any combination of up to and including t erroneous symbols, or locate and correct up to and including ⌊t/2⌋ erroneous symbols at unknown locations. As an erasure code, it can correct up to and including t erasures at locations that are known and provided to the algorithm, or it can detect and correct combinations of errors and erasures. Reed–Solomon codes are also suitable as multiple-burst bit-error correcting codes, since a sequence of b + 1 consecutive bit errors can affect at most two symbols of size b. The choice of t is up to the designer of the code and may be selected within wide limits.
There are two basic types of Reed–Solomon codes – original view and BCH view – with BCH view being the most common, as BCH view decoders are faster and require less working storage than original view decoders.
PCS
PCS is the Physical Coding Sublayer. The PCS layer defines how Ethernet frames are encoded into groups of bits called “blocks” that are transmitted via the PMA/PMD layer.
Summary
So if you see any increase of PCS and some of the Lane you can start to look at layer 1. That could be problem with QSFP or any of the cabling in between.
Traffic Loopback.
I found another command, traffic-loopback source system device mac under the interface. What this does is loop the traffic on the QSFP. What is the reason for that? The port on the side you put this on will go down then UP, the other side will go down. The next this is that EOS will send small amount of traffic that it sends out too QSFP and it will loop it back in to the port. The EOS trick it selfs to think it sending traffic and receive some traffic back.
What I used that for is to see if the QSFP is faulty or the physical port, if you then see PCS BER increasing then is not the cabling, it’s the QSFP or the port. If you change the QSFP and still see the same problem, create a TAC case with all this findings and let them also have a look.