====== EAPS Troubleshooting ====== ===== Log ===== ==== Jan 12th ==== * After restarting both **Server_Room** and **5th_Floor** I could not get link between the switches to come up on 10G reducing speed to 1G brings the link up but produces a log CRC errors and the connection is drop once every 10-20 secs even when using 1G sftps * Any fiber run that is dropping seems to have some sort of CRC errors * run from **Ag_Room** to **OLD_KSU** giving CRC errors on **OLD_KSU** port 1:53 even after restarting both switches reconfigured both ports to 1G and CRC errors stopped * After reconfiguring both ports back to 10G the CRC errors seemed to have stopped * Run from **server room** to **5th floor** stayed down after restarting both switches. Link did come up after reconfiguring both ports to 1G, both both 1:31 on server room switch and 1:54 on 5th floor switch had CRC errors * As of 8pm on Friday all other switches showing 0 CRC errors * Fibers in **Ag_Room** are transposed from the port descriptions. ==== Jan 13th ==== * Over the night of the 12th of Jan EAPS master reporting on **Office** and **OLD_KSU** switches * reduced speed of **Ag_room** and **OLD_KSU** switches to 1G seeing if CRC errors are causing on EAPS master * **Office** switch is having port 1:54 lose connection every hour or so, even after multiple restarts * **Elementary_LD** port 1:54 had no issues until about (over 12hrs) 7am on Jan 13th then started loosing uplink every 1-2 min over a 5-10min period. Started again at 10:30am * **Elementary_LD** was restarted and port 1:54 resumed normal operation * Switched **Office** to **Elementary_LD** and **5th_Floor** to **Server_Room** over to using 1G sftp on the front of each switch and reconfigured EAPS ring to use different ports as of **12:01pm** * single mode fiber link between **Elementary_LD** to **Business_AD_RM** started dropping a lot reducing speed to 1G to see if that stops the dropping **12:20pm** * **12:25pm** After reducing both links to 1G only one side would get a connection reverted to 10G and rebooted **Elementary_LD** and **Business_AD_RM** * **4:30pm** Pulling sftps off the ftp+ module seems to be fixing the issues. But still having issues between **OLD_KSU** to **5th_Floor** planning on fixing moving link to front sftp ports * It's starting to look like we have two bad ftp+ modules in **5th_Floor** and **Elementary_LD** ==== Jan 14th ==== * **10am** moved other sftp from ftp+ module to front sftp ports on both **5th_Floor** and **OLD_KSU** switches * **3pm** still getting about 1 drops/hr on **OLD_KSU** to **Ag_Room** link and 4-6 drops a hr on **Elementary_LD** to **Business_AD** all other links are staying up * **6pm** it looks like we have a bad ftp+ module in **5th_Floor**, **Elementary_LD** and a failing module in **OLD_KSU** or **Ag_Room** ==== Jan 15th ==== * **7:30am** moved other sftp from ftp+ module to front sftp ports on both **Ag_Room** and **OLD_KSU** switches * **8:30am** link between **Ag_Room** and **Business_AD** started dropping * **8:51am** moved back to 10G sftp+ on **OLD_KSU** for **Ag_Room** link * **9:00am** moved other sftp from ftp+ module to front sftp ports on both **Ag_Room** and **Business_AD** * **11:00am** it looks like only one link is dropping: 1:54 on **Elementary_LD** * **11:30am** replaced 10G card in **Elementary_LD** * **1:00pm** only one drop at old **KSU_ROOM** at 12:35pm ===== Switch Addresses ===== ^Address^Name^ |02:04:96:83:48:AB|Server_Room| |00:04:96:83:9F:20|ITV_CLoset| |00:04:96:83:9F:57|Office| |00:04:96:83:9F:62|Elementary_Ed| |00:04:96:83:9F:7E|Business_AD_RM| |00:04:96:83:9F:82|Ag_Room| |00:04:96:83:9F:7B|Old_KSU_Closet| |00:04:96:83:9F:64|5th_Floor| ===== Ring Topology ===== ^Server_Room^^ |1:31|Link_To_5th_Floor| |2:31|Link_To_ITV_Closet| ^ITV_CLoset^^ |1:53|Link_To_Server_Room| |1:54|Link_To_Office| ^Office^^ |1:53|Link_to_ITV_Room| |1:54|Link_to_Elementary_Ed_closet| ^Elementary_Ed^^ |1:53|Link_To_Office| |1:54|Link_To_Business_AD_Room| ^Business_AD_RM^^ |1:53|Link_To_Elementary_Ed_Closet| |1:54|Link_To_Ag_Room| ^Ag_Room^^ |1:53|Link_To_Business_AD_RM| |1:54|Link_To_Old_KSU_RM| ^Old_KSU_Closet^^ |1:53|Link_To_Ag_RM| |1:54|Link_To_5TH_Floor| ^5th_Floor^^ |1:53|Link_To_Old_KSU| |1:54|Link_TO_Server_RM| ===== Useful Commands ===== show ports 1:53-1:54 configuration no-refresh configure ports 1:53 auto off speed 1000 duplex full show ports 1:53-1:54 congestion show log | include "Received Link-Down-Pdu" show log | include show ports 1:53,1:54 rxerrors show ports 1:31,2:31 rxerrors show switch show version ===== EAPS Down Example ===== Slot-1 Office.1 # show eaps "seb-eaps" Name: seb-eaps Priority: Normal State: Links-Down Running: Yes Enabled: Yes Mode: Transit Primary port: 1:53 Port status: Up Tag status: Tagged Secondary port: 1:54 Port status: Down Tag status: Tagged Slot-1 Elementary_Ed.1 # show eaps "seb-eaps" Name: seb-eaps Priority: Normal State: Links-Down Running: Yes Enabled: Yes Mode: Transit Primary port: 1:53 Port status: Down Tag status: Tagged Secondary port: 1:54 Port status: Up Tag status: Tagged ===== Packet Loss ===== --- 10.1.1.77 ping statistics --- 3365 packets transmitted, 3328 received, +11 errors, 1% packet loss, time 3443267ms rtt min/avg/max/mdev = 0.185/0.722/1024.694/17.753 ms, pipe 4 --- 10.1.3.14 ping statistics --- 3368 packets transmitted, 3333 received, +11 errors, 1% packet loss, time 3445370ms rtt min/avg/max/mdev = 0.207/1.040/1016.357/17.592 ms, pipe 4 ==== Example Rx error report ==== show ports 1:53,1:54 rxerrors no-refresh Port Rx Error Monitor Port Link Rx Rx Rx Rx Rx Rx Rx State Crc Over Under Frag Jabber Align Lost ================================================================================ 1:53 A 0 0 0 0 0 0 0 1:54 A 0 0 0 0 0 0 0 ================================================================================ Link State: A-Active, R-Ready, NP-Port Not Present L-Loopback